notebooks/worksheet1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Worksheet 1: File locations and pre-processing\n",
    "\n",
    "The exercises in this worksheet demonstrate some of the tools available for data analysis, and how to prepare CORDEX output for analysis (pre-processing). This can be time consuming for large amounts of data, so in this worksheet a small subset of data is used to easily demonstrate the steps involved.\n",
    "\n",
    "[CORDEX data](https://cordex.org/data-access/how-to-access-the-data/) is in [NetCDF](https://www.unidata.ucar.edu/software/netcdf/) format (a standard format in climate science) so that it can be used in post processing packages such as Python and the Python library [Iris](http://scitools.org.uk/iris/docs/latest/index.html).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-warning\">\n",
    "<b>By the end of this worksheet you should be able to:</b><br> \n",
    "- Identify and list the names of CORDEX output data in netCDF format using standard Linux commands.<br>\n",
    "- Use basic Iris commands to load data files, and view Iris cubes. <br>\n",
    "- Use Iris commands to merge netCDF files\n",
    "- Take a subset of the data based on a date range\n",
    "- Save the output as NetCDF files.\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Note:</b> In the boxes where there is code or where you are asked to type code, click in the box, then press <kbd>Ctrl</kbd> + <kbd>Enter</kbd> to run the code. <br>\n",
    "<b>Note:</b> An percentage sign <code>%</code> is needed to run some commands on the shell. It is noted where this is needed.<br>\n",
    "<b>Note:</b> A hash <code>#</code> denotes a comment; anything written after this character does not affect the command being run. <br>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents\n",
    "\n",
    "### [1.1: Data locations and file names](#1.1)\n",
    "\n",
    "### [1.2: Getting started with Python and Iris](#1.2)\n",
    "\n",
    "### [1.3: Merge Problems](#1.3)\n",
    "\n",
    "### [1.4 Extracting data within a specific time range](#1.4)\n",
    "\n",
    "### [1.5: Saving data to a new file](#1.5)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='1.1'></a>\n",
    "\n",
    "## 1.1 Data locations and file names\n",
    "\n",
    "The datasets used within these worksheets are already linked to the notebook in order to provide quick and easy access for the purpose of this training. However the commands learned in this worksheet provide useful context for future work in a linux and unix scripting environment.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**a)** Firstly, find out what location you are currently in by using the **`pwd`** command; **`pwd`** stands for **print working directory**.\n",
    "\n",
    "In the cell below type **`%pwd`** on a new line and then press <kbd>Ctrl</kbd> + <kbd>Enter</kbd>.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Type %pwd below and press 'ctrl' + 'enter'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b)** List the contents of this directory; **`ls`** stands for **list** and using the **`-l`** option gives a longer listing with more information, such as file size and modification date.\n",
    "\n",
    "In the cell below type **`%ls`** on a separate line then press <kbd>Ctrl</kbd> + <kbd>Enter</kbd>.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Type %ls and press 'ctrl' + 'enter'.\n",
    "\n",
    "\n",
    "# Type %ls -l and press 'ctrl' + 'enter'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "<div class=\"alert alert-block alert-success\">\n",
    "<b>Question:</b> What is the difference between <code>ls</code> and <code>ls -l</code>?  What extra information do you see? Which file was edited most recently?\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Answer</b>: <br>\n",
    "_...Double click here to type your answer..._\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c)** To avoid conflicts of data when running locally, we will take a copy of the source files used in the training (this is not needed if running on the cloud, we instead download from an S3 bucket). Run the command in the following cell. It might take a few minutes to complete.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import subprocess\n",
    "\n",
    "subprocess.run([\"rsync\", \"-r\", \"/project/ciid/projects/PRECIS/worksheets/data_v2\", \".\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the command works correctly you should see the message:\n",
    "\n",
    "`CompletedProcess(args=['rsync', '-r', '/project/ciid/projects/PRECIS/worksheets/data_v2', '.'], returncode=0)`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**d)** Move to the directory (i.e. folder) called `data_v2/EAS-22`. This directory contains CORDEX data for the East Asia Domain.\n",
    "\n",
    "**Hint:** The `cd` command stands for _change directory_\n",
    "\n",
    "Type your command(s) below and press <kbd>Ctrl</kbd> + <kbd>Enter</kbd>.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Type your %cd [directory-path] command below and press 'ctrl' + 'enter'.\n",
    "\n",
    "\n",
    "# List the contents of this directory, using a previous command."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**d)** There are a lot of files in this directory! The file names contain information on the simulated date of the data they contain - you'll learn more about the naming convention for CORDEX data in another presention.\n",
    "\n",
    "For now, list only the files containing monthly temperature data using the following command:\n",
    "\n",
    "Type **`%ls tas*mon*`** in the code block below.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Type %ls tas*mon* and press 'ctrl' + 'enter'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Note:</b> The asterisk character <code>*</code> (also known as <i>glob</i>) matches any string within the filename\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**e)** This still returns too many files to comfortably count manually. **`wc`** stands for **word count**; combining this command with **`ls`** allows us to count the number of items in that directory.\n",
    "\n",
    "In the cell below type **`%ls tas*mon* | wc -l`** then press <kbd>Ctrl</kbd> + <kbd>Enter</kbd>.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Type %ls *mon* | wc -l command below and press 'ctrl' + 'enter'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "<div class=\"alert alert-block alert-success\">\n",
    "<b>Question:</b> How many nc files are in this directory, in total?\n",
    "<br>How many of these nc files contain the string 'historical'; relating to the historical climate simulation? What command do you need to use to find this out?\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Answer</b>:\n",
    "<br>_Total number of nc files:\n",
    "<br>Number of historical nc files:\n",
    "<br>Command used to find number of historical nc files:_\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**f)** To list all the files containing monthly data from a period starting 202101, we use the code **`*mon_202101-??????.nc`**\n",
    "\n",
    "Type below **`%ls *mon_202101-??????.nc`**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Type %ls *mon_202101-??????.nc and press 'ctrl' + 'enter'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Note:</b> The question mark character <code>?</code> matches any single character\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**g)** Now move up one level in the directory tree and list the directories.\n",
    "\n",
    "Type `cd ..` to move up one level in the directory tree and press <kbd>Ctrl</kbd> + <kbd>Enter</kbd> to execute the command\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Type %cd .. and press 'ctrl' + 'enter'."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "<div class=\"alert alert-block alert-success\">\n",
    "<b>Question:</b> Which directory are you in now?  What else can you see in this directory?\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Answer</b>:<br>\n",
    "_...Double click here to type your answer..._\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "<a id='1.2'></a>\n",
    "\n",
    "## 1.2 Getting started with Python and Iris\n",
    "\n",
    "<p><img src=\"img/python_and_iris.png\" alt=\"python + iris logo\" style=\"float: center; height: 100px;\"/></p>\n",
    "\n",
    "Python is a general purpose programming language. Python supports modules and packages, which encourages program modularity and code reuse.\n",
    "\n",
    "We also use the Python library [Iris](http://scitools.org.uk/iris/docs/v2.4.0/index.html), which is written in Python and is maintained by the Met Office. Iris seeks to provide a powerful, easy to use, and community-driven Python library for analysing and visualising meteorological and oceanographic data sets.\n",
    "\n",
    "The top level object in Iris is called a <b>cube</b>. A cube contains data and metadata about a phenomenon (i.e. air_temperature). Iris handles several different types of file formats, loading them into Iris cubes.\n",
    "\n",
    "For a brief introduction to Iris and the cube formatting please read this Introduction page here:\n",
    "\n",
    "http://scitools.org.uk/iris/docs/v2.4.0/userguide/iris_cubes.html\n",
    "\n",
    "For future reference please refer to the Iris website:\n",
    "\n",
    "http://scitools.org.uk/iris/docs/v2.4.0/index.html\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**a)** First, run the code-blocks below to **load** a file into Iris and **print** its metadata structure. <br>\n",
    "\n",
    "To run the code, click in the box below and press <kbd>Ctrl</kbd> + <kbd>Enter</kbd>.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd \"/home/h03/fris/code/PyPRECIS/notebooks/data_v2\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the necessary modules\n",
    "import iris\n",
    "import glob\n",
    "import os\n",
    "\n",
    "# turn off future warnings in Iris for readability\n",
    "import warnings\n",
    "\n",
    "warnings.simplefilter(action=\"ignore\", category=FutureWarning)\n",
    "\n",
    "# this is needed so that the plots are generated inline with the code instead of a separate window\n",
    "%matplotlib inline\n",
    "\n",
    "# Provide the path of your sample data\n",
    "data_dir = \"/home/h03/fris/code/PyPRECIS/notebooks/data_v2\"\n",
    "domain = \"EAS-22\"\n",
    "\n",
    "path_dir = os.path.join(data_dir, domain)\n",
    "sample_data = path_dir + \"/sample_data.nc\"\n",
    "\n",
    "# Constraint the reading to a single variable and load it into an Iris cube\n",
    "cube = iris.load_cube(sample_data)\n",
    "\n",
    "# Print the Iris cube\n",
    "print(cube)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "<div class=\"alert alert-block alert-success\">\n",
    "<b>Question:</b> Can you explain how our sample data we printed above relates to this picture?\n",
    "<img src=\"img/multi_array_to_cube.png\" alt=\"diagram of an Iris cube\" style=\"height: 300px\"/> <br>\n",
    "\n",
    "- Is our data above a 3D or a 2D cube? <br>\n",
    "- What are the cube dimensions? <br>\n",
    "- How many grid boxes is the latitudinal range divided into? <br>\n",
    "- What meteorological variable does this cube represent? <br>\n",
    "- What unit is used for this variable? <br>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Double click here to type your answer:</b>:\n",
    "<br>_Is this cube 3D or 2D?\n",
    "<br>What are the cube dimensions?\n",
    "<br>How many grid boxes is the latitudinal range divided into?\n",
    "<br>What meteorological variable does this cube represent?\n",
    "<br>What unit is used for this variable?_\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b)** Now **plot** the data for the selected variable: <br>\n",
    "\n",
    "To run the code, click in the box below and press <kbd>Ctrl</kbd> + <kbd>Enter</kbd>.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import iris.quickplot as qplt\n",
    "\n",
    "plt.figure(figsize=(12, 12))  # Set the figure size\n",
    "\n",
    "# find the edges of grid cells (Aka the boounds)\n",
    "cube.coord(\"grid_latitude\").guess_bounds()\n",
    "cube.coord(\"grid_longitude\").guess_bounds()\n",
    "\n",
    "qplt.pcolormesh(cube)  # Plot the cube\n",
    "plt.title(cube.name())  # Add plot title\n",
    "plt.clim(0, 5e-4)  # Set colour bar range\n",
    "plt.show()  # Show the plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "Sense check the data plotted:\n",
    "\n",
    "- Can you make out sections of the East Asian coastline?\n",
    "- How about the scale?\n",
    "\n",
    "As we progress through these workbooks, we'll learn how to process the data into more intuitive units and mask / add country boundaries, so it's easier to understand the information.\n",
    "\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='1.3'></a>\n",
    "\n",
    "## 1.3 Merge problems\n",
    "\n",
    "**a)** When using data, we want a single cube for all fields with the same standard name and sequential timesteps. `iris.load` will return as few cubes as possible, by collecting cubes from multiple files together. However, on some occasions this merge process does not give a single cube when we would expect it to.\n",
    "\n",
    "This section demonstrates how to deal with cases like this and make a single cube for these data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "You can read more about iris loading behaviour <a href=\"https://scitools.org.uk/iris/docs/v2.4.0/userguide/loading_iris_cubes.html\">here</a>.\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As an example we will load some temperature data from the [East Asia Domain](https://cordex.org/domains/region-7-east-asia/) at 25 km resolution\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# these variables form part of the standard CORDEX filename convention\n",
    "rcm = \"GERICS-REMO2015\"\n",
    "experiment = \"historical\"\n",
    "gcm = \"NCC-NorESM1-M\"\n",
    "variable = \"tas\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's find out which of the files match the pattern above.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_name = variable + \"*\" + gcm + \"*\" + experiment + \"*\" + rcm + \"*\"\n",
    "file_list = glob.glob(path_dir + \"/\" + file_name)\n",
    "\n",
    "# Complete the print statement to see which files match the specified criteria\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "How many files were returned?    \n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Double click here to type your answer:**\n",
    "\n",
    "How many files were returned?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b)** Run the line of code below to try and force Iris to load this data into a _single_ cube. \n",
    "<div class=\"alert alert-block alert-success\">\n",
    "<b>This command will return an error</b>, read the output to find out why.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cube = iris.load_cube(file_list)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The important part of this error is the following message:\n",
    "\n",
    "    ConstraintMismatchError: failed to merge into a single cube.\n",
    "      cube.attributes values differ for keys: 'history', 'creation_date', 'tracking_id'\n",
    "\n",
    "\n",
    "Instead we will load this data with `iris.load()` and then look more closely at the data, before we fix the issues.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cubes = iris.load(file_list)\n",
    "print(cubes)\n",
    "print()\n",
    "for cube in cubes:\n",
    "    print(cube.attributes[\"creation_date\"], cube.attributes[\"tracking_id\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "<b>Question:\n",
    "\n",
    "- How many cubes are in the cube list you loaded?<br>\n",
    "- Are they all the same size in space?<br>\n",
    "- Do they have the same number of timesteps? Why do you think this is? (Hint: look again at the filenames we are loading)\n",
    "- What are the differences in the attributes and do you think this is important when analysing your data?\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Answer</b>:\n",
    "<br>_How many cubes are in the cube list you loaded?\n",
    "<br>Are they all the same size in space?\n",
    "<br>Do they have the same number of timesteps? Why do you think this is? (Hint: look again at the filenames we are loading)\n",
    "<br>What are the differences in the attributes?\n",
    "<br>Do you think these differences are important when analysing your data?_\n",
    "\n",
    "---\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c)** Now let's solve this problem so we can get a single cube. We will do this using the [equalise_attributes](https://scitools.org.uk/iris/docs/v2.4.0/iris/iris/experimental/equalise_cubes.html) function from Iris.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from iris.util import equalise_attributes\n",
    "\n",
    "equalise_attributes(cubes)\n",
    "\n",
    "# now print the attributes of each cube\n",
    "for cube in cubes:\n",
    "    print(list(cube.attributes.keys()))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "\n",
    "The equalise_attributes function has removed the metadata which is inconsistent between the cubes.\n",
    "\n",
    "<b>Question:\n",
    "\n",
    "- Why might it be a bad idea to apply this function without looking at the data first?\n",
    "</b>\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Type your answer:</b>:\n",
    "<br>_Why might it be a bad idea to apply this function without looking at the data first?_\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following loop is an alternative method to eliminate the mismatching attributes\n",
    "\n",
    "    for i, icube in enumerate(cube):\n",
    "        del cube[i].attributes['creation_date']\n",
    "        del cube[i].attributes['tracking_id']\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**d)** We can now merge the data into a single cube.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cube = cubes.concatenate_cube()\n",
    "print(cube)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-success\">\n",
    "<li> Now that we have combined multiple files into a single cube, what is the cube's shape? \n",
    "<li> How does this compare with the cube_list created in 1.2e? \n",
    "<li> Based on all the information you've gained about the data so far, what time period do you expect the data in this cube to span?\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Type your answers here**\n",
    "\n",
    "- Now that we have combined multiple files into a single cube, what is the cube's shape?\n",
    "- How does this compare with the cube_list created in 1.2e?\n",
    "- Based on all the information you've gained about the data so far, what time period do you expect the data in this cube to span?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='1.4'></a>\n",
    "\n",
    "## 1.4 Extracting data within a specific time range\n",
    "\n",
    "**a)** This is a lot of data, and so for now, we will cut this down to include December 1989 to November 1991 inclusive using a time constraint. Edit the code below to specify the missing end date.\n",
    "(**Hint:** specify the adjacent months BEFORE and AFTER the time period you wish to keep.)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from iris.time import PartialDateTime\n",
    "\n",
    "print(\"original cube first and last dates\")\n",
    "print(cube.coord(\"time\")[0])\n",
    "print(cube.coord(\"time\")[-1])\n",
    "\n",
    "time_constraint = iris.Constraint(\n",
    "    time=lambda cell: PartialDateTime(year=1989, month=11)\n",
    "    < cell.point\n",
    "    < PartialDateTime(year=YYYY, month=MM)\n",
    ")\n",
    "sub_cube = cube.extract(time_constraint)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b)** Check the first and last timesteps in your constrained cube are correct:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print()\n",
    "print(\"new cube first and last dates\")\n",
    "print(sub_cube.coord(\"time\")[0])\n",
    "print(sub_cube.coord(\"time\")[-1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id='1.5'></a>\n",
    "\n",
    "## 1.5 Save data to a new file\n",
    "\n",
    "**a)** We will now save this data to a new file.\n",
    "\n",
    "Take note of the file names. Well chosen filenames can help you keep track of the contents of your files. We suggest developing a consistent syntax based on the filename patterns of the CORDEX data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "out_file_name = (\n",
    "    variable\n",
    "    + \"_\"\n",
    "    + domain\n",
    "    + \"_\"\n",
    "    + gcm\n",
    "    + \"_\"\n",
    "    + experiment\n",
    "    + \"_r1ip1_\"\n",
    "    + rcm\n",
    "    + \"_v2_mon_198912-199111.nc\"\n",
    ")\n",
    "\n",
    "save_location = data_dir + \"cordex_training/\"\n",
    "\n",
    "%mkdir {save_location}\n",
    "\n",
    "print(\"saving file to: \" + save_location + out_file_name)\n",
    "iris.save(sub_cube, save_location + out_file_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Note:</b>\n",
    "As we progress through these worksheets, keep a note of how we update the file names when making further changes to the data.\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "\n",
    "<div class=\"alert alert-block alert-success\">\n",
    "<b>b) Question:</b> Use the <b>cd</b> and <b>ls</b> commands to check the NetCDF directory that you have been creating new files in. <br>\n",
    "\n",
    "- Confirm the names of the new files you have been creating. <br>\n",
    "- What is the size of the concatenated file (containing December 1979 to November 1989 data)?\n",
    "</div>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use %cd and %ls to list the contents of your new directory containing NetCDF files:\n",
    "\n",
    "\n",
    "# use %ls -lh to compare the size of the original files and final netcdf file you saved"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<b>Answer:</b><br>\n",
    "_Size of the file written out at the end:_\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<center>\n",
    "<div class=\"alert alert-block alert-warning\">\n",
    "<b>This completes worksheet 1.</b> <br>You have created pre-processed files (metdata fixed, concenated over time, extracted data in a specific time range and saved in NetCDF format). <br>\n",
    "In worksheet 2, you will begin to analyse these files.\n",
    "</div>\n",
    "</center>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p><img src=\"img/MO_MASTER_black_mono_for_light_backg_RBG.png\" alt=\"Met Office Logo\" style=\"float: center; height: 100px;\"/></p>\n",
    "<center>© Crown Copyright, Met Office. All rights reserved.</center>\n",
    "<center>This file is part of PyPrecis and is released under the BSD 3-Clause license.</center>\n",
    "<center>See LICENSE in the root of the repository for full licensing details.</center>\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  },
  "vscode": {
   "interpreter": {
    "hash": "30fc2c9368d5cc76c4fad6f63328738b6cf0a0091c34cff8da6b063602b9d1b0"
   }
  },
  "widgets": {
   "state": {},
   "version": "1.1.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}