{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using FluxEngine with in situ data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set the working directory\n",
    "Before we do anything, we need to set the working directory to the root FluxEngine directory so that we know the relative location of the commandline tools we'll be using (just as we did in the previous tutorial). Again, it doesn't matter if you don't understand this code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Working directory is now: /home/rr/Files/fluxengine_v3/FluxEngine\n"
     ]
    }
   ],
   "source": [
    "#This code changes the working directory from the tutorial folder to the root FluxEngine folder.\n",
    "import os\n",
    "if os.path.basename(os.getcwd()) != \"FluxEngine\":\n",
    "    os.chdir(os.path.join(\"..\",\"..\"))\n",
    "print \"Working directory is now:\", os.getcwd()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Workflow for working with in situ data\n",
    "\n",
    "The diagram below describes the workflow for working with in situ CO<sub>2</sub> data with FluxEngine, and consists of four steps. This assumes in situ data is provided as a text formatted file, for example tab or comma seperated value files.\n",
    "<img src=\"images/insitu_workflow2.png\" width=\"400\">\n",
    "\n",
    "<b>Step 1)</b> In situ fugacity of CO<sub>2</sub> (fCO<sub>2</sub>) measurements are often made from differing depths and can make use of different instrument setups. FluxEngine provides a tool which enables fCO<sub>2</sub> measurements to be reanalysed to a consistent temperature and depth field.\n",
    "\n",
    "<b>Step 2)</b> Next the data must be converted into the gridded netCDF file format that FluxEngine uses. The `text2ncdf.py` tool can perform this conversion, and will automatically bin data into a specified grid resolution (calculating the mean and standard deviations where more than one data point falls into a single grid cell).\n",
    "\n",
    "<b>Step 3)</b> With these netCDF files, FluxEngine can calculate the gas fluxes. This step uses the `ofluxghg_run.py` tool we've used in the previous tutorial.\n",
    "\n",
    "<b>Step 4)</b> You'll now have calculated gas fluxes from the in situ data but they will be in a gridded format, rather than the original text format of the in situ data. FluxEngine provides a tool to convert netCDF output files into text files (`ncdf2text.py`) and another tool to directly append FluxEngine output variables to an existing text file (such as the original in situ data file), `append2insitu.py`.\n",
    "\n",
    "This tutorial will demonstrate this process step by step."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The data\n",
    "We'll be working with the data collected from the AMT 2013 cruise. <b>TODO: describe data, give link and reference, plot cruise track...<\\b> https://doi.pangaea.de/10.1594/PANGAEA.878492\n",
    "\n",
    "<img src=\"images/kitidis_track.png\" width=\"400\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1) Reanalysis CO<sub>2</sub> data to a consistent temperature and depth\n",
    "The reanalysis tool can be run as a standalone commandline tool or by importing it as a Python module and including in a custom script. We will run it using the commandline tool. First, lets use the build-in help function to query the tool and see what options it provides."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "usage: reanalyse_socat_driver.py [-h] [-socat_dir SOCAT_DIR]\r\n",
      "                                 [-socat_files [SOCAT_FILES [SOCAT_FILES ...]]]\r\n",
      "                                 [-sst_dir SST_DIR] [-sst_tail SST_TAIL]\r\n",
      "                                 [-output_dir OUTPUT_DIR] [-startyr STARTYR]\r\n",
      "                                 [-endyr ENDYR]\r\n",
      "                                 [-regions [<keyword_list> [<keyword_list> ...]]]\r\n",
      "                                 [-withcoastal WITHCOASTAL] [-asciioutput]\r\n",
      "                                 [-useaatsr] [-usereynolds] [-keepduplicates]\r\n",
      "                                 [-keeptempfiles] [-socatversion SOCATVERSION]\r\n",
      "                                 [-notsocatformat] [-year_col YEAR_COL]\r\n",
      "                                 [-month_col MONTH_COL] [-day_col DAY_COL]\r\n",
      "                                 [-hour_col HOUR_COL] [-minute_col MINUTE_COL]\r\n",
      "                                 [-second_col SECOND_COL]\r\n",
      "                                 [-longitude_col LONGITUDE_COL]\r\n",
      "                                 [-latitude_col LATITUDE_COL]\r\n",
      "                                 [-salinity_col SALINITY_COL]\r\n",
      "                                 [-salinity_sub_col SALINITY_SUB_COL]\r\n",
      "                                 [-SST_C_col SST_C_COL] [-Tequ_col TEQU_COL]\r\n",
      "                                 [-air_pressure_col AIR_PRESSURE_COL]\r\n",
      "                                 [-air_pressure_sub_col AIR_PRESSURE_SUB_COL]\r\n",
      "                                 [-air_pressure_equ_col AIR_PRESSURE_EQU_COL]\r\n",
      "                                 [-fCO2_col FCO2_COL]\r\n",
      "                                 [-expocode_col EXPOCODE_COL]\r\n",
      "\r\n",
      "Utility which allows reanalysis of SOCAT data to a consistent temperature and\r\n",
      "depth. (under continual development, use with care) This tool uses previously\r\n",
      "developed code by ???. AUTHOR Python driver script by Tom Holding\r\n",
      "<t.m.holding@exeter.ac.uk> 2018 Everything else is from the reanalyse_socat\r\n",
      "open source project (see the reanalyse_socat directory in the FE tools\r\n",
      "folder).\r\n",
      "\r\n",
      "optional arguments:\r\n",
      "  -h, --help            show this help message and exit\r\n",
      "\r\n",
      "Data paths:\r\n",
      "  -socat_dir SOCAT_DIR  Path to the directory containing tsv formatted SOCAT\r\n",
      "                        data.\r\n",
      "  -socat_files [SOCAT_FILES [SOCAT_FILES ...]]\r\n",
      "                        A list of region codes to process (e.g. NA for North\r\n",
      "                        Atlantic). If no region codes are specified the global\r\n",
      "                        database is assumed.\r\n",
      "  -sst_dir SST_DIR      Path to the directory containing netCDF formatted sea\r\n",
      "                        sureface temperature (either Reynolds or AATSR).\r\n",
      "  -sst_tail SST_TAIL    The SST file name which follows yyyymm. Do not include\r\n",
      "                        the year or month digits.\r\n",
      "  -output_dir OUTPUT_DIR\r\n",
      "                        Path to the output directory\r\n",
      "\r\n",
      "Run period:\r\n",
      "  -startyr STARTYR      The year to start importing SOCAT data for.\r\n",
      "  -endyr ENDYR          The last year to importing SOCAT data for (includes\r\n",
      "                        this year).\r\n",
      "\r\n",
      "Optional settings:\r\n",
      "  -regions [<keyword_list> [<keyword_list> ...]]\r\n",
      "                        A list of region codes to process. These will be used\r\n",
      "                        to name output files and should correspond to the list\r\n",
      "                        of file names given in -socat_files. If no region code\r\n",
      "                        is specified it is assumed a single file containing\r\n",
      "                        global data is provided.\r\n",
      "  -withcoastal WITHCOASTAL\r\n",
      "                        The region code (defined using -regions) which\r\n",
      "                        corresponds to coastal data. Coastal data is appended\r\n",
      "                        to other regions where gridcells overlap and any\r\n",
      "                        remaining data is analysed seperately. Do not specify\r\n",
      "                        if no coastal data is used. Default is None (no\r\n",
      "                        coastal data used).\r\n",
      "  -asciioutput          To output data as ascii lists rather than gridded\r\n",
      "                        netcdf.\r\n",
      "  -useaatsr             To use the AATSR SST data.\r\n",
      "  -usereynolds          To use the Reynolds SST data.\r\n",
      "  -keepduplicates       Do not detect and remove duplicate data points.\r\n",
      "  -keeptempfiles        Do not delete the temporary (full output) files\r\n",
      "                        produced by the reanalyse_socat.\r\n",
      "\r\n",
      "Mode (socat vs non-socat):\r\n",
      "  -socatversion SOCATVERSION\r\n",
      "                        If running in SOCAT mode this determines the version\r\n",
      "                        of the SOCAT data files to read\r\n",
      "  -notsocatformat       Set if running the tool on data which is not formatted\r\n",
      "                        exactly as socat files are. This allows custom column\r\n",
      "                        names/values to be set using the other options in this\r\n",
      "                        group (see below). Note that no filtering based on\r\n",
      "                        expedition code or quality control flags will be\r\n",
      "                        conducted in this run mode.\r\n",
      "  -year_col YEAR_COL    Column name or number (indexed from 0) of the column\r\n",
      "                        which specifies year.\r\n",
      "  -month_col MONTH_COL  Column name or number (indexed from 0) of the column\r\n",
      "                        which specifies month.\r\n",
      "  -day_col DAY_COL      Column name or number (indexed from 0) of the column\r\n",
      "                        which specifies day of the month.\r\n",
      "  -hour_col HOUR_COL    Column name or number (indexed from 0) of the column\r\n",
      "                        which specifies hours.\r\n",
      "  -minute_col MINUTE_COL\r\n",
      "                        Column name or number (indexed from 0) of the column\r\n",
      "                        which specifies minutes.\r\n",
      "  -second_col SECOND_COL\r\n",
      "                        Column name or number (indexed from 0) of the column\r\n",
      "                        which specifies seconds.\r\n",
      "  -longitude_col LONGITUDE_COL\r\n",
      "                        Column name or number (indexed from 0) of the column\r\n",
      "                        which specifies longitude [0, 360).\r\n",
      "  -latitude_col LATITUDE_COL\r\n",
      "                        Column name or number (indexed from 0) of the column\r\n",
      "                        which specifies latitude (-90, 90).\r\n",
      "  -salinity_col SALINITY_COL\r\n",
      "                        Column name or number (indexed from 0) of the column\r\n",
      "                        which specifies salinity.\r\n",
      "  -salinity_sub_col SALINITY_SUB_COL\r\n",
      "                        Column name or number (indexed from 0) of the column\r\n",
      "                        which specifies values which will be used to\r\n",
      "                        substitute missing salinity values (e.g. modelled\r\n",
      "                        salinity field). If not specified then the value 35\r\n",
      "                        will be substituted.\r\n",
      "  -SST_C_col SST_C_COL  Column name or number (indexed from 0) of the column\r\n",
      "                        which specifies sea surface temperature (sub-skin) in\r\n",
      "                        Censius.\r\n",
      "  -Tequ_col TEQU_COL    Column name or number (indexed from 0) of the column\r\n",
      "                        which specifies water temperature at equlibrium in\r\n",
      "                        Celsius.\r\n",
      "  -air_pressure_col AIR_PRESSURE_COL\r\n",
      "                        Column name or number (indexed from 0) of the column\r\n",
      "                        which specifies air pressure.\r\n",
      "  -air_pressure_sub_col AIR_PRESSURE_SUB_COL\r\n",
      "                        Column name or number (indexed from 0) of the column\r\n",
      "                        which specifies values which will be used to substitue\r\n",
      "                        missing air pressure values (e.g. modelled air\r\n",
      "                        pressure).\r\n",
      "  -air_pressure_equ_col AIR_PRESSURE_EQU_COL\r\n",
      "                        Column name or number (indexed from 0) of the column\r\n",
      "                        which specifies air pressure at equilibrium.\r\n",
      "  -fCO2_col FCO2_COL    Column name or number (indexed from 0) of the column\r\n",
      "                        which specifies fugacity of carbon dioxide.\r\n",
      "  -expocode_col EXPOCODE_COL\r\n",
      "                        Column name or number (indexed from 0) of the column\r\n",
      "                        which specifies a cruise or dataset specific code\r\n",
      "                        (e.g. 'Expocode' in SOCAT datasets).\r\n",
      "\r\n",
      "Both this script and the FluxEngine are in continual development. Use with\r\n",
      "care.\r\n"
     ]
    }
   ],
   "source": [
    "!python fluxengine_src/tools/reanalyse_socat_driver.py -h"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that this follows the same syntax when we ran FluxEngine using the commandline tool in the previous tutorial; the `python fluxengine_src/tools/reanalyse_socat_driver.py` command tells the computer to run the reanalysis tool using Python, and then we supply a list of options (in this case a single option, `-h`, asking for the help information). This `-h` option ('h' standing for 'help') can be used with any of the FluxEngine commandline tools to view a built in description of the tool and its options.\n",
    "\n",
    "Looking through the help message shows a lot of options, many of which are optional, or only occasionally used. Other options are always required, such as `socat_dir` and `-socat_files` which define the path to the input data (the word 'socat' is an artefact of the original use of the tool to reanalyse the SOCAT database), `-output_dir` which defines the directory that our reanalysed files will be saved to, and `-sst_dir` and `-sst_tail` which defined the path to a sea surface temperature climatology which will be used as the consistent temperature field. \n",
    "\n",
    "The tool is designed to be easily used with data that follow the format and column names of the SOCAT dataset, but our in situ data file isn't formatted exactly like this. We'll therefore also use the `-notsocatformat` option and specify the names of each of the required columns for time and date, longitude and latitude, salinity, in situ SST, air pressure and fCO<sub>2</sub>. The resulting command is long, but it's not complicated. We call the run the `reanalyse_socat_driver.py` tool using `python` and pass a series of options specifying the settings discussed above. Try running it now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "ATTENTION: selected to use Reynolds SST. This assumes that the sstdir points to the Reynolds data (if not you need to change it).\n",
      "\n",
      "No interpolation option specified so will exit at interpolation stage.\n",
      "Reading in data from SOCAT file: /home/rr/Files/fluxengine_v3/FluxEngine/Tutorials/02_using_insitu_data/insitu_data/kitidis_consistent.tsv\n",
      "This can take a while for large datasets.\n",
      "\n",
      "Subsetting data for year range: 2013 2013\n",
      "num duplicates is:  0\n",
      "/home/rr/anaconda2/lib/python2.7/site-packages/numpy/core/records.py:513: FutureWarning: Numpy has detected that you may be viewing or writing to an array returned by selecting multiple fields in a structured array. \n",
      "\n",
      "This code may break in numpy 1.15 because this will return a view instead of a copy -- see release notes for details.\n",
      "  return obj.view(dtype=(self.dtype.type, obj.dtype))\n",
      "Writing to: /home/rr/Files/fluxengine_v3/FluxEngine/fluxengine_src/tools/reanalyse_socat/output/reanalysed_data/10/kitidis_from_2013_to_2013_10_custom_insitu.txt\n",
      "Writing to: /home/rr/Files/fluxengine_v3/FluxEngine/fluxengine_src/tools/reanalyse_socat/output/reanalysed_data/11/kitidis_from_2013_to_2013_11_custom_insitu.txt\n",
      "Total number of data points imported and used from file: /home/rr/Files/fluxengine_v3/FluxEngine/Tutorials/02_using_insitu_data/insitu_data/kitidis_consistent.tsv : 3677\n",
      "Have finished the pre-interpolation (reanalysis) stage.\n",
      "If you are going to run with post-interpolation, please ensure that interpolated data files are named as expected and in the expected directories.\n",
      "Reanalysis completed successfully.\n"
     ]
    }
   ],
   "source": [
    "!python fluxengine_src/tools/reanalyse_socat_driver.py -socat_dir \"Tutorials/02_using_insitu_data/insitu_data\" -socat_files \"kitidis_consistent.tsv\" -output_dir \"output/reanalysed/kitidis/\" -usereynolds -sst_dir \"Tutorials/02_using_insitu_data/reynolds_avhrr_only_monthly_calculated_tmh\" -sst_tail \"01_OCF-SST-GLO-1M-100-REYNOLDS_TMH.nc\" -startyr 2013 -endyr 2013 -regions \"kitidis\" -asciioutput -notsocatformat -year_col \"year\" -month_col \"month\" -day_col \"day\" -hour_col \"hour\" -minute_col \"minute\" -second_col \"second\" -longitude_col \"lon\" -latitude_col \"lat\" -salinity_col \"salinity\" -salinity_sub_col \"salinity_cl\" -SST_C_col \"temperature\" -Tequ_col \"temperature_eq\" -air_pressure_col \"air_pressure\" -air_pressure_sub_col \"air_pressure_cl\" -air_pressure_equ_col \"air_pressure_equ\" -fCO2_col \"fCO2water\" -expocode_col 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For convenience, lets perform one more step and merge the reanalysed fCO2 and SST climatology data into a single file. This is entirely optional but makes our work easier because we only need to perform the next steps on one file. The following Python script uses a library called Pandas to take the reanalysed data from October and November (this covers the entire period of the cruise) and joins them into one dataset, then adds the reanalysed data for temperature, partial pressure and fugacity into the original in situ data file. This combined file is then saved as a seperate file called `kitidis_reanalysed.tsv` which we will continue to work with in the next steps.\n",
    "\n",
    "Don't worry if you don't understand the specifics of the Python code below. You could equally well do this step by hand in Excel, or by another programming language like R or Matlab."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Reanalysed data successfully merged.\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd #allows using data frames\n",
    "import numpy as np #matrix manipulation\n",
    "\n",
    "#read the original in situ data and the reanalysed data.\n",
    "#Note that the in situ data only covers October and November.\n",
    "insituDf = pd.read_table(\"Tutorials/02_using_insitu_data/insitu_data/kitidis_consistent.tsv\", sep=\"\\t\")\n",
    "reanOct = pd.read_table(\"output/reanalysed/kitidis/output/reanalysed_data/10/kitidis_from_2013_to_2013_10_custom_insitu.txt\", sep=\",\")\n",
    "reanNov = pd.read_table(\"output/reanalysed/kitidis/output/reanalysed_data/11/kitidis_from_2013_to_2013_11_custom_insitu.txt\", sep=\",\")\n",
    "\n",
    "#Create and populate columns with each of the variables we want from the reanalysed data\n",
    "#The first part of each column uses data from October and the second part uses data from November\n",
    "#to create a continuous time series which is consistent with the in situ file.\n",
    "SST_C = np.empty(len(insituDf)); #in situ SST\n",
    "SST_C[0:len(reanOct)] = reanOct[\"SST_C\"]\n",
    "SST_C[len(reanOct):] = reanNov[\"SST_C\"]\n",
    "\n",
    "Tcl_C = np.empty(len(insituDf)); #consistent SST field data\n",
    "Tcl_C[0:len(reanOct)] = reanOct[\"Tcl_C\"]\n",
    "Tcl_C[len(reanOct):] = reanNov[\"Tcl_C\"]\n",
    "\n",
    "fCO2_SST = np.empty(len(insituDf)); #in situ fCO2\n",
    "fCO2_SST[0:len(reanOct)] = reanOct[\"fCO2_SST\"]\n",
    "fCO2_SST[len(reanOct):] = reanNov[\"fCO2_SST\"]\n",
    "\n",
    "fCO2_Tym = np.empty(len(insituDf)); #reanalysied fCO2\n",
    "fCO2_Tym[0:len(reanOct)] = reanOct[\"fCO2_Tym\"]\n",
    "fCO2_Tym[len(reanOct):] = reanNov[\"fCO2_Tym\"]\n",
    "\n",
    "pCO2_SST = np.empty(len(insituDf)); #in situ pCO2\n",
    "pCO2_SST[0:len(reanOct)] = reanOct[\"pCO2_SST\"]\n",
    "pCO2_SST[len(reanOct):] = reanNov[\"pCO2_SST\"]\n",
    "\n",
    "pCO2_Tym = np.empty(len(insituDf)); #reanalysed fCO2\n",
    "pCO2_Tym[0:len(reanOct)] = reanOct[\"pCO2_Tym\"]\n",
    "pCO2_Tym[len(reanOct):] = reanNov[\"pCO2_Tym\"]\n",
    "\n",
    "#Add the reanalysed columns to the in situ data.\n",
    "insituDf[\"SST_C\"] = SST_C\n",
    "insituDf[\"Tcl_C\"] = Tcl_C\n",
    "insituDf[\"fCO2_SST\"] = fCO2_SST\n",
    "insituDf[\"fCO2_Tym\"] = fCO2_Tym\n",
    "insituDf[\"pCO2_SST\"] = pCO2_SST\n",
    "insituDf[\"pCO2_Tym\"] = pCO2_Tym\n",
    "\n",
    "#Write merged dataframe to file\n",
    "f = open(\"output/reanalysed/kitidis/kitidis_reanalysed.tsv\", mode='w')\n",
    "insituDf.to_csv(f, header=True, index=False, sep=\"\\t\", float_format='%.3f')\n",
    "f.close();\n",
    "print \"Reanalysed data successfully merged.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2) Converting to netCDF format\n",
    "FluxEngine requires input data in netCDF format which means we need to convert the text formated data into netCDF file(s). As before FluxEngine provides two ways to do this; the `text2ncdf.py` tool can be imported as a Python module or ran as a standalone commandline tool.\n",
    "\n",
    "We'll continue using the commandline method. The tool requires a list of files to convert to netCDF (for us, this is just the single file we created in step 1), a start and stop time, a list of columns to convert into netCDF variables and output file path. The tool works by matching in situ measurements to a temporal and spatial grid, then calculating the mean value of all the measurements that are in a particular grid. Additionally the name (or index) of the date/time, longitude and latitude columns are also required because the tool uses this information to match in situ measurements to grid cells. Finally the spatial and temporal resolution must be specified. For our purposes we'll analyse the whole cruise together as a single time point, so we set the temporal resolution to 60 days (enough to span the period of the cruise), and the default spatial resolution of 1x1 degree will be more than adequate.\n",
    "\n",
    "For more information on these settings you can view the help information by running the `text2ncdf.py` with the `-h` option. Run the command below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parsing command line arguments.\n",
      "Calculating dimensions.\n",
      "The following columns will be extracted:\n",
      "\ttemperature\n",
      "\ttemperature_eq\n",
      "\twindspeed\n",
      "\twind_moment2\n",
      "\twind_moment3\n",
      "\tice_coverage\n",
      "\tair_pressure\n",
      "\tair_pressure_equ\n",
      "\tair_pressure_cl\n",
      "\tsalinity\n",
      "\tsalinity_cl\n",
      "\txCO2air\n",
      "\tfCO2water\n",
      "\tSST_C\n",
      "\tTcl_C\n",
      "\tfCO2_SST\n",
      "\tfCO2_Tym\n",
      "\tpCO2_SST\n",
      "\tpCO2_Tym\n",
      "Processing data in file output/reanalysed/kitidis/kitidis_reanalysed.tsv\n",
      "Writing output netCDF file(s)...\n",
      "Finished converting text file to netCDF3. There were 0 values which fell outside the specified lat/lon or start/stop time boundaries.\n",
      "Number of missing values found: 6\n"
     ]
    }
   ],
   "source": [
    "!python fluxengine_src/tools/text2ncdf.py \"output/reanalysed/kitidis/kitidis_reanalysed.tsv\" --startTime \"2013-10-01 00:00:00\" --endTime \"2013-11-15 00:00:00\" --ncOutPath \"output/reanalysed/kitidis/kitidis_reanalysed.nc\" --temporalResolution \"60 00:00\" --dateIndex 7 --cols 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 --latProd \"lat\" --lonProd \"lon\""
   ]
  },
  {
   "attachments": {
    "viewing_converted_netcdf.png": {
     "image/png": "[encoded data, 231184 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If all goes well you'll see a message saying \"Finished converting text file to netCDF3\". You might see a message telling you there were 6 missing values. This isn't anything to worry about. <b>TODO: Why? Where do the missing values come from?? Explain!</b>\n",
    "\n",
    "Lets open the netCDF file we've just created in Panoply to view its contents. You can find it in `output/reanalysed/kitidis/kitidis_reanalysed.nc`. You'll see that there are count, mean and standard deviation variables for each of the columns we converted. The count data stores the number of data points which were used to calculate the mean and standard deviation for each grid cell. Double click a few of the variables to plot them and check that they look sensible.\n",
    "\n",
    "![viewing_converted_netcdf.png](attachment:viewing_converted_netcdf.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3) Running FluxEngine\n",
    "We have our reanalysed data in netCDF format, now we can use FluxEngine to calculate the air-sea CO<sub>2</sub> fluxes!\n",
    "\n",
    "We need to create a configuration file to tell FluxEngine to use our new netCDF file as input. For convenience this has been created for you (see the file stores at `Tutorials/02_using_insitu_data/kitidis_cruise_W14_reanalysed.conf`). The only significant difference between this configuration file, and the one you created in the first tutorial is the selected input data and the temporal resolution.\n",
    "\n",
    "<i>Temporal resolution</i> - We want FluxEngine to calculate fluxes for the whole cruise track as a single file rather than as seperate monthly (or some other time step) files. To do this we use the `temporal_resolution` option, setting its value to `60 00:00` which indicates a timestep of 60 days, 0 hours and 0 minutes. This is a large enough interval of time to encompase the entire cruise, and so only a single time point will be generated.\n",
    "\n",
    "<i>Input data</i> - Open the configuration file (`Tutorials/02_using_insitu_data/kitidis_cruise_W14_reanalysed.conf`) and scroll down to the section where input data is specified. You can see that for every input we're using the netCDF file we generated in step 2 and providing a different product name to access different variables within the netCDF file for the different inputs. Notice as well that we're using the reanalysed version of the fCO2 data (`fCO2_Tym_mean`) along with the sea surface temperature climatology (`Tcl_C_mean`).\n",
    "\n",
    "Looking at the other settings you will see that we're using the 'rapid' flux equation and the Wanninkhof 2014 gas transfer parameterisation and Schmidt number parameterisation.\n",
    "\n",
    "Lets calculate the air-sea CO<sub>2</sub> fluxes using the commandline tool we used in the previous tutorial:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing on 'rr-HP-Compaq-Elite-8300-MT' at 22/05/2019 13:08:22\n",
      "Parsing settings file at: /home/rr/Files/fluxengine_v3/FluxEngine/fluxengine_src/settings.xml\n",
      "(ofluxghg_flux_calc, FluxEngine._load_lon_lat_time): Grid dimensions set to: (180, 360)\n",
      "Converting sstfnd from celsius to kelvin.\n",
      "/home/rr/Files/fluxengine_v3/FluxEngine/fluxengine_src/fe_core.py:1039: RuntimeWarning: invalid value encountered in less\n",
      "  self.data[\"pco2_sw\"].fdata[abs(self.data[\"pco2_sw\"].fdata)<0.1]=DataLayer.missing_value#IGA_SOCATv4\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using SSTfnd data selection with correction for skin temperature (SSTskin = SSTfnd - 0)(ignoring SSTskin data in configuration file).\n",
      "/home/rr/Files/fluxengine_v3/FluxEngine/fluxengine_src/fe_core.py:1263: RuntimeWarning: invalid value encountered in greater\n",
      "  if npany(self.data[\"pressure\"].fdata > 10000.0):\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the RAPID model (from Woolf et al., 2016)\n",
      "(rate_parameterisation.py: k_Wanninkhof2014.__call__) Using the Wanninkhof 2014 k parameterisation\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SUCCESS writing file /home/rr/Files/fluxengine_v3/FluxEngine/output/kitidis_cruise/kitidis_W14_reanalysed.nc\n",
      "Flux engine exited with exit code: 0\n",
      "01 Oct 2013 00:00:00 completed successfully.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!python ofluxghg_run.py \"Tutorials/02_using_insitu_data/kitidis_cruise_W14_reanalysed.conf\" -s \"2013-10-01\" -e \"2013-11-15\" -l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hopefully you'll see a \"completed successfully\" message at the end. Don't worry if you see some `RuntimeWarning`s about invalid values being encountered. This is because there are a few stray NaNs (not a number) in the data. FluxEngine will just ignore these and it won't affect the calculation.\n",
    "\n",
    "Use Panoply to open the output netCDF file located in `output/kitidis_cruise/kitidis_W14_reanalysed.nc` and check that the output looks reasonable. Use Panoply to plot the air-sea flux (OF) variable, and the gas transfer velocity variable. Take the time to explore the output and visualise any others you might be interested in."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4) Combining FluxEngine output with in situ data and visualising\n",
    "So far so good; we're reanalysed the in situ data fCO<sub>2</sub> data to a consistent temperature and depth, we've created a netCDF file containing our input data and used FluxEngine to calculate the gas fluxes. However, in many situtations calculating gas fluxes will be only one part of a wider analysis and it would be useful to extract variables from the FluxEngine output and put them back into our original in situ data file. Lets do this now.\n",
    "\n",
    "As before, FluxEngine provides two way to do this; importing a module into a custom Python script or using a commandline tool, and as before we will use the commandline tool. The tool we'll use is called `append2insitu.py`, so lets run it with the `-h` help command to see how to use it.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "usage: append2insitu.py [-h] [--varsToAppend VARSTOAPPEND [VARSTOAPPEND ...]]\r\n",
      "                        [--delim DELIM] [--latCol LATCOL] [--lonCol LONCOL]\r\n",
      "                        [-m MISSINGVALUE] [--encoding ENCODING] [-d DATEINDEX]\r\n",
      "                        [-n NUMCOMMENTLINES [NUMCOMMENTLINES ...]]\r\n",
      "                        feOutput insituData outPath\r\n",
      "\r\n",
      "Converts netCDF3 data produced as output from FluxEngine into text data (e.g.\r\n",
      "csv, tsv). The text output will contain one column for each data layer, and\r\n",
      "one row for each cell of the grid (if there is no missing data). One text file\r\n",
      "will be created for each netCDF3 file.\r\n",
      "\r\n",
      "positional arguments:\r\n",
      "  feOutput              path to a FluxEngine output file\r\n",
      "  insituData            path to a plain text in situ data file\r\n",
      "  outPath               Path to store the produced text file.\r\n",
      "\r\n",
      "optional arguments:\r\n",
      "  -h, --help            show this help message and exit\r\n",
      "  --varsToAppend VARSTOAPPEND [VARSTOAPPEND ...]\r\n",
      "                        List of netCDF variable names which will be appended\r\n",
      "                        to the in situ data file. Default is the air-sea gas\r\n",
      "                        flux (OF) and the gas transfer velocity (K)\r\n",
      "  --delim DELIM         delimiter token used to seperate data in the in situ\r\n",
      "                        data file. Default is a tab.\r\n",
      "  --latCol LATCOL       Name or column number (starting from 0) of the\r\n",
      "                        latitude column in the in situ input file. Default is\r\n",
      "                        a 'latitude'.\r\n",
      "  --lonCol LONCOL       Name or column number (starting from 0) of the\r\n",
      "                        longitude column in the in situ input file. Default is\r\n",
      "                        'longitude'.\r\n",
      "  -m MISSINGVALUE, --missingValue MISSINGVALUE\r\n",
      "                        Value used to indicate a missing value in the input\r\n",
      "                        text file. Default is 'nan'.\r\n",
      "  --encoding ENCODING   Encoding of the input text file. Default it 'utf-8'\r\n",
      "  -d DATEINDEX, --dateIndex DATEINDEX\r\n",
      "                        The column number (starting from 0) which contains the\r\n",
      "                        date/time field in the in situ input data file.\r\n",
      "                        Default is a 0.\r\n",
      "  -n NUMCOMMENTLINES [NUMCOMMENTLINES ...], --numCommentLines NUMCOMMENTLINES [NUMCOMMENTLINES ...]\r\n",
      "                        Number of comment lines before the header in your in\r\n",
      "                        situ input data file. These lines will be ignored).\r\n",
      "                        Default is a 0.\r\n"
     ]
    }
   ],
   "source": [
    "!python fluxengine_src/tools/append2insitu.py -h"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These help files can take some practice to understand. Where it says 'positional arguments' it lists three mandatory pieces of information that the tool needs to run; `feOutput` which specifies the path to the FluxEngine output file, `insituData` which specifies the path to the in situ data file, and `outPath` which specifies path and name of the output file that will be created.\n",
    "\n",
    "Next there is a list of 'optional arguments'. These are additional pieces of information that you can specify. They will often have default values (stated in the description of each optional argument) and specifying a different value simply overwrites the default. We'll use the `--varsToAppend` option to choose which FluxEngine output variables we would like to append to the in situ data. The tool uses the date, time and longitude and latitude fields to match up data from the in situ file and the FluxEngine output. We will therefore use the `--dateIndex` option to tell the tool which column in the in situ data contains the date information, and the `--lonCol` and `--latCol` options to specify the longitude and latitude column names.\n",
    "\n",
    "Lets construct our commandline command and run it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python fluxengine_src/tools/append2insitu.py \"output/kitidis_cruise/kitidis_W14_reanalysed.nc\" \"output/reanalysed/kitidis/kitidis_reanalysed.tsv\" \"output/kitidis_cruise/kitidis_W14_reanalysed_merged.tsv\" --varsToAppend \"OF\" \"OK3\" \"OSFC\" \"OIC1\" --dateIndex 7 --lonCol \"lon\" --latCol \"lat\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see a new text data file located here `output/kitidis_cruise/kitidis_W14_reanalysed_merged.tsv` that contains the original in situ data, the reanalysed fCO2 and pCO2 along with the SST climatology used to calculate it, and the FluxEngine outputs for air-sea flux (OF), gas transfer velocity (OK3), aqueous concentration of CO2 (OSFC) and interface concentration of CO2 (OIC1).\n",
    "\n",
    "With all this data together in one file we can easily visualise it! The python code below will produce some time series plots, showing temperature (in situ and the consistent temperature field), wind speed, molar fraction of atmospheric CO<sub>2</sub> (xCO2), sub-skin and interface concentration of CO2 and the sea-to-air CO2 flux. Don't worry if you don't understand the Python code below - you could visualise the data using any visualisation tools you like. The important thing is to understand the main steps we've taken."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 129240 characters]\n",
      "text/plain": [
       "<Figure size 720x936 with 7 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from datetime import datetime\n",
    "\n",
    "\n",
    "#Returns time elapsed between baselineDate and date in days (fractions of days allowed).\n",
    "def calc_fraction_days(baselineDate, date):\n",
    "    delta = date-baselineDate;\n",
    "    fractionDays = delta.days + (delta.seconds) / (24*60*60.0);\n",
    "    return fractionDays;\n",
    "\n",
    "#Time series plots\n",
    "#Make a copy of the data kitidis\n",
    "data = pd.read_table(\"output/kitidis_cruise/kitidis_W14_reanalysed_merged.tsv\", sep=\"\\t\", parse_dates=[7])\n",
    "\n",
    "#Rename columns to remove units for matplotlib to function correctly\n",
    "newColNamesMap = {colName:colName.split('[')[0].strip() for colName in data.keys()};\n",
    "data = data.rename(index=str, columns=newColNamesMap);\n",
    "\n",
    "#Calculate time in fractions of days\n",
    "baselineDate = datetime(2013, 10, 9); #First day of data in the kitidis dataset\n",
    "time = [calc_fraction_days(baselineDate, d) for d in data[\"datetime\"]];\n",
    "data[\"fraction_days\"] = time;\n",
    "initialDiscard = 750; #Note that there's a problem with the xCO2 data. This just ignore the first section of the data for purely cosmetic reasons.\n",
    "time = np.array(time) - time[initialDiscard];\n",
    "time = time[initialDiscard:];\n",
    "\n",
    "#Plotting settings\n",
    "pad = 10; #padding between y axis and labels\n",
    "rot = 90; #y label rotation\n",
    "labelsize = 14;\n",
    "ticksize = 14;\n",
    "\n",
    "alpha = 0.85;\n",
    "diffColour = (0.4, 0.4, 0.4);\n",
    "\n",
    "f, (ax1, ax2, ax3, ax4, ax5, ax6, ax7) = plt.subplots(7, sharex=True, sharey=False, figsize=[10,13]);\n",
    "plt.xlabel(\"time (days since 16-10-2013)\", fontsize=labelsize);\n",
    "plt.tick_params(labelsize=ticksize);\n",
    "\n",
    "ax1.plot(time, data[\"temperature\"][initialDiscard:], 'k', linewidth=2, alpha=alpha, label=\"in situ\"); #Temp [°C]\n",
    "ax1.plot(time, data[\"Tcl_C\"][initialDiscard:], 'b', linewidth=2, alpha=alpha, label=\"satellite\"); #Temp [°C]\n",
    "ax1.set_ylabel(\"SST\\n($^{\\circ}C$)\", labelpad=pad, rotation=rot, fontsize=labelsize, horizontalalignment='center');\n",
    "ax1.tick_params(labelsize=ticksize);\n",
    "ax1.grid(True);    \n",
    "    \n",
    "ax2.plot(time, data[\"windspeed\"][initialDiscard:], 'b', linewidth=2, alpha=alpha); #windspeed [m s-1]\n",
    "ax2.set_ylabel(\"wind speed\\n($ms^{-1}$)\", labelpad=pad, rotation=rot, fontsize=labelsize, horizontalalignment='center');\n",
    "ax2.tick_params(labelsize=ticksize);\n",
    "ax2.grid(True);\n",
    "\n",
    "ax3.plot(time, data[\"xCO2air\"][initialDiscard:], 'b', linewidth=2, alpha=alpha); #xCO2air_interp [µmol/mol]\n",
    "ax3.set_ylabel(\"xCO$_{2}$\\n($\\mu mol/mol$)\", labelpad=pad, rotation=rot, fontsize=labelsize, horizontalalignment='center');\n",
    "ax3.tick_params(labelsize=ticksize);\n",
    "ax3.grid(True);\n",
    "    \n",
    "ax4.plot(time, data[\"fCO2_SST\"][initialDiscard:], 'k', linewidth=2, alpha=alpha, label=\"in situ\"); #fCO2water_SST_wet [µatm]\n",
    "ax4.plot(time, data[\"fCO2_Tym\"][initialDiscard:], 'b', linewidth=2, alpha=alpha, label=\"reanalysed\"); #fCO2water_SST_wet [µatm]\n",
    "ax4.set_ylabel(\"$fCO_{2}$\\n($\\mu atm$)\", labelpad=pad, rotation=rot, fontsize=labelsize, horizontalalignment='center');\n",
    "ax4.tick_params(labelsize=ticksize);\n",
    "ax4.grid(True);\n",
    "\n",
    "ax5.plot(time, data[\"OSFC\"][initialDiscard:], 'b', linewidth=2, alpha=alpha, label=\"reanalysed\"); #CO2 flux\n",
    "ax5.set_ylabel(\"sub-skin\\nconcentration\\n$CO_{2}$ ($gCm^{-3}$)\", labelpad=pad, rotation=rot, fontsize=labelsize, horizontalalignment='center');\n",
    "ax5.tick_params(labelsize=ticksize);\n",
    "ax5.grid(True);\n",
    "    \n",
    "ax6.plot(time, data[\"OIC1\"][initialDiscard:], 'b', linewidth=2, alpha=alpha, label=\"reanalysed\"); #CO2 flux\n",
    "ax6.set_ylabel(\"interface\\nconcentration\\n$CO_{2}$ ($gCm^{-3}$)\", labelpad=pad, rotation=rot, fontsize=labelsize, horizontalalignment='center');\n",
    "ax6.tick_params(labelsize=ticksize);\n",
    "ax6.grid(True);\n",
    "\n",
    "ax7.plot(time, data[\"OF\"][initialDiscard:], 'b', linewidth=2, alpha=alpha, label=\"reanalysed\"); #CO2 flux\n",
    "ax7.set_ylabel(\"$CO_{2}$ flux\\n($gCm^{-2}day^{-1}$)\", labelpad=pad, rotation=rot, fontsize=labelsize, horizontalalignment='center');\n",
    "ax7.tick_params(labelsize=ticksize);\n",
    "ax7.grid(True);\n",
    "    \n",
    "plt.tight_layout();\n",
    "plt.subplots_adjust(hspace = 0.1);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps\n",
    "These first two tutorials have covered the basics and you should be able to use the configuration files, commandline tools and workflow described in them as the basis to run your own air-sea gas flux calculations. There are three more tutorials which cover different aspects. You can do them in any order depending on your interest (and you probably won't be interested enough to do all of them).\n",
    "\n",
    " - Calculating air-sea CO<sub>2</sub> fluxes from a fixes station in the Baltic Sea\n",
    " - Calculating air-sea fluxes for N<sub>2</sub>O for an AMT cruise.\n",
    " - <b>Something else. Swedish lakes and freshwater stuff?</b>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
