{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Jupyter notebooks\n",
    "This workshop uses Jupyter notebooks. This is a convenient way to combined text and images with code. You'll find Python code examples embedded in the text tutorial. You can run these examples by clicking the run (play) button. The code can also be edited and ran again so that you can see the consequences of changes to the code. To start again from the beginning you can reset the entire notebook by choosing `Kernel` from the top menu, then `Restart & clear output`.\n",
    "\n",
    "When you first open a Jupyter notebook the working directory is set to the directory containing the Jupyter notebook. For most of the exercises in this tutorial we'll want the working directory to be set to the root FluxEngine directory, which is one directory up. Set the working directory by running the code below (don't worry if you don't understand it, the important thing is that you run it at the start):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Working directory is now: /home/rr/Files/fluxengine_v3/FluxEngine\n"
     ]
    }
   ],
   "source": [
    "#This code changes the working directory from the tutorial folder to the root FluxEngine folder.\n",
    "import os\n",
    "if os.path.basename(os.getcwd()) != \"FluxEngine\":\n",
    "    os.chdir(os.path.join(\"..\",\"..\"))\n",
    "print \"Working directory is now:\", os.getcwd()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Getting setup with FluxEngine\n",
    "Hopefully you have already downloaded and installed FluxEngine prior to this workshop. It you haven't you should do this now. To download the latest version go to the git hub repository [here](https://github.com/oceanflux-ghg/FluxEngine/).\n",
    "\n",
    "FluxEngine uses several third party libraries and tools. A script is included in the download to automatically install these dependencies for MacOS and Linux, and installation instructions are included for Windows in section 3 of [the instructions](https://github.com/oceanflux-ghg/FluxEngine/blob/master/FluxEngineV3_instructions.pdf). If you haven't done this already you should do this now.\n",
    "\n",
    "**TODO: provide description to run installer for mac/linux. Discuss installation of anaconda**\n",
    "**TODO: seperate reset working directory functions? Use a library to manage these?**\n",
    "**TODO: Box explaining the difference between python, commandline and ipython commands? Describe what the commandline is and how to run a commandline command in Windows/Mac**\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Verifying FluxEngine has been installed correctly\n",
    "It is important to verify that FluxEngine, and its dependencies, have been installed correctly. This will prevent problems arising later, and gives us confidence that the flux calculation is being performed correctly by running a known scenario and comparing it to previously published data. This is all handled automatically using a verification script. If you have already ran the verification script when you first installed FluxEngine you do not have to do this again. If you haven't already, then do this by running the code below. It will take about ten minutes to run, but you can continue with the rest of the tutorial while you wait, remembering to check back to make sure you got a message saying \"Validation successful! All values are within threshold limits\" when it has finished."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/rr/Files/Tasks/20190408_SummerSchool/FluxEngine\n",
      "Running FluxEngine for year 2010...\n",
      "Executing on 'rr-HP-Compaq-Elite-8300-MT' at 20/05/2019 11:17:24\n",
      "Parsing settings file at: /home/rr/Files/Tasks/20190408_SummerSchool/FluxEngine/fluxengine_src/settings.xml\n",
      "(ofluxghg_flux_calc, FluxEngine._load_lon_lat_time): Grid dimensions set to: (180, 360)\n",
      "Preprocessing rain: Flipping longitude orientation.\n",
      "Preprocessing rain: Flipping latitude orientation.\n",
      "Assuming correct orientation for pco2_sw. Variable has not been flipped.\n",
      "Preprocessing pco2_sw: Flipping longitude orientation.\n",
      "Assuming correct orientation for vco2_air. Variable has not been flipped.\n",
      "Preprocessing vco2_air: Flipping longitude orientation.\n",
      "Assuming correct orientation for salinity. Variable has not been flipped.\n",
      "Converting pressure from Pa to mbar\n",
      "Assuming correct orientation for pco2_sst. Variable has not been flipped.\n",
      "Preprocessing pco2_sst: Flipping longitude orientation.\n",
      "Converting pco2_sst from kelvin to celsius.\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the saline skin model (0.100000 psu added to skin salinities)\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SST gradient handling is on, using SSTskin and SSTfnd = SSTskin + 0 for flux calculation (ignoring SSTfnd data in configuration file).\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the RAPID model (from Woolf et al., 2016)\n",
      "(rate_parameterisation.py: k_Nightingale2000.__call__) Using the Nightingale et al., 2000 (N00) k parameterisation\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SUCCESS writing file /home/rr/Files/Tasks/20190408_SummerSchool/FluxEngine/output/validate_socatv4_sst_salinity_N00/2010/01/OceanFluxGHG-month01-jan-2010-v0.nc\n",
      "Flux engine exited with exit code: 0\n",
      "01 Jan 2010 00:00:00 completed successfully.\n",
      "\n",
      "Parsing settings file at: /home/rr/Files/Tasks/20190408_SummerSchool/FluxEngine/fluxengine_src/settings.xml\n",
      "(ofluxghg_flux_calc, FluxEngine._load_lon_lat_time): Grid dimensions set to: (180, 360)\n",
      "Preprocessing rain: Flipping longitude orientation.\n",
      "Preprocessing rain: Flipping latitude orientation.\n",
      "Assuming correct orientation for pco2_sw. Variable has not been flipped.\n",
      "Preprocessing pco2_sw: Flipping longitude orientation.\n",
      "Assuming correct orientation for vco2_air. Variable has not been flipped.\n",
      "Preprocessing vco2_air: Flipping longitude orientation.\n",
      "Assuming correct orientation for salinity. Variable has not been flipped.\n",
      "Converting pressure from Pa to mbar\n",
      "Assuming correct orientation for pco2_sst. Variable has not been flipped.\n",
      "Preprocessing pco2_sst: Flipping longitude orientation.\n",
      "Converting pco2_sst from kelvin to celsius.\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the saline skin model (0.100000 psu added to skin salinities)\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SST gradient handling is on, using SSTskin and SSTfnd = SSTskin + 0 for flux calculation (ignoring SSTfnd data in configuration file).\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the RAPID model (from Woolf et al., 2016)\n",
      "(rate_parameterisation.py: k_Nightingale2000.__call__) Using the Nightingale et al., 2000 (N00) k parameterisation\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SUCCESS writing file /home/rr/Files/Tasks/20190408_SummerSchool/FluxEngine/output/validate_socatv4_sst_salinity_N00/2010/02/OceanFluxGHG-month02-feb-2010-v0.nc\n",
      "Flux engine exited with exit code: 0\n",
      "01 Feb 2010 00:00:00 completed successfully.\n",
      "\n",
      "Parsing settings file at: /home/rr/Files/Tasks/20190408_SummerSchool/FluxEngine/fluxengine_src/settings.xml\n",
      "(ofluxghg_flux_calc, FluxEngine._load_lon_lat_time): Grid dimensions set to: (180, 360)\n",
      "Preprocessing rain: Flipping longitude orientation.\n",
      "Preprocessing rain: Flipping latitude orientation.\n",
      "Assuming correct orientation for pco2_sw. Variable has not been flipped.\n",
      "Preprocessing pco2_sw: Flipping longitude orientation.\n",
      "Assuming correct orientation for vco2_air. Variable has not been flipped.\n",
      "Preprocessing vco2_air: Flipping longitude orientation.\n",
      "Assuming correct orientation for salinity. Variable has not been flipped.\n",
      "Converting pressure from Pa to mbar\n",
      "Assuming correct orientation for pco2_sst. Variable has not been flipped.\n",
      "Preprocessing pco2_sst: Flipping longitude orientation.\n",
      "Converting pco2_sst from kelvin to celsius.\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the saline skin model (0.100000 psu added to skin salinities)\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SST gradient handling is on, using SSTskin and SSTfnd = SSTskin + 0 for flux calculation (ignoring SSTfnd data in configuration file).\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the RAPID model (from Woolf et al., 2016)\n",
      "(rate_parameterisation.py: k_Nightingale2000.__call__) Using the Nightingale et al., 2000 (N00) k parameterisation\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SUCCESS writing file /home/rr/Files/Tasks/20190408_SummerSchool/FluxEngine/output/validate_socatv4_sst_salinity_N00/2010/03/OceanFluxGHG-month03-mar-2010-v0.nc\n",
      "Flux engine exited with exit code: 0\n",
      "01 Mar 2010 00:00:00 completed successfully.\n",
      "\n",
      "Parsing settings file at: /home/rr/Files/Tasks/20190408_SummerSchool/FluxEngine/fluxengine_src/settings.xml\n",
      "(ofluxghg_flux_calc, FluxEngine._load_lon_lat_time): Grid dimensions set to: (180, 360)\n",
      "Preprocessing rain: Flipping longitude orientation.\n",
      "Preprocessing rain: Flipping latitude orientation.\n",
      "Assuming correct orientation for pco2_sw. Variable has not been flipped.\n",
      "Preprocessing pco2_sw: Flipping longitude orientation.\n",
      "Assuming correct orientation for vco2_air. Variable has not been flipped.\n",
      "Preprocessing vco2_air: Flipping longitude orientation.\n",
      "Assuming correct orientation for salinity. Variable has not been flipped.\n",
      "Converting pressure from Pa to mbar\n",
      "Assuming correct orientation for pco2_sst. Variable has not been flipped.\n",
      "Preprocessing pco2_sst: Flipping longitude orientation.\n",
      "Converting pco2_sst from kelvin to celsius.\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the saline skin model (0.100000 psu added to skin salinities)\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SST gradient handling is on, using SSTskin and SSTfnd = SSTskin + 0 for flux calculation (ignoring SSTfnd data in configuration file).\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the RAPID model (from Woolf et al., 2016)\n",
      "(rate_parameterisation.py: k_Nightingale2000.__call__) Using the Nightingale et al., 2000 (N00) k parameterisation\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SUCCESS writing file /home/rr/Files/Tasks/20190408_SummerSchool/FluxEngine/output/validate_socatv4_sst_salinity_N00/2010/04/OceanFluxGHG-month04-apr-2010-v0.nc\n",
      "Flux engine exited with exit code: 0\n",
      "01 Apr 2010 00:00:00 completed successfully.\n",
      "\n",
      "Parsing settings file at: /home/rr/Files/Tasks/20190408_SummerSchool/FluxEngine/fluxengine_src/settings.xml\n",
      "(ofluxghg_flux_calc, FluxEngine._load_lon_lat_time): Grid dimensions set to: (180, 360)\n",
      "Preprocessing rain: Flipping longitude orientation.\n",
      "Preprocessing rain: Flipping latitude orientation.\n",
      "Assuming correct orientation for pco2_sw. Variable has not been flipped.\n",
      "Preprocessing pco2_sw: Flipping longitude orientation.\n",
      "Assuming correct orientation for vco2_air. Variable has not been flipped.\n",
      "Preprocessing vco2_air: Flipping longitude orientation.\n",
      "Assuming correct orientation for salinity. Variable has not been flipped.\n",
      "Converting pressure from Pa to mbar\n",
      "Assuming correct orientation for pco2_sst. Variable has not been flipped.\n",
      "Preprocessing pco2_sst: Flipping longitude orientation.\n",
      "Converting pco2_sst from kelvin to celsius.\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the saline skin model (0.100000 psu added to skin salinities)\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SST gradient handling is on, using SSTskin and SSTfnd = SSTskin + 0 for flux calculation (ignoring SSTfnd data in configuration file).\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the RAPID model (from Woolf et al., 2016)\n",
      "(rate_parameterisation.py: k_Nightingale2000.__call__) Using the Nightingale et al., 2000 (N00) k parameterisation\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SUCCESS writing file /home/rr/Files/Tasks/20190408_SummerSchool/FluxEngine/output/validate_socatv4_sst_salinity_N00/2010/05/OceanFluxGHG-month05-may-2010-v0.nc\n",
      "Flux engine exited with exit code: 0\n",
      "01 May 2010 00:00:00 completed successfully.\n",
      "\n",
      "Parsing settings file at: /home/rr/Files/Tasks/20190408_SummerSchool/FluxEngine/fluxengine_src/settings.xml\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(ofluxghg_flux_calc, FluxEngine._load_lon_lat_time): Grid dimensions set to: (180, 360)\n",
      "Preprocessing rain: Flipping longitude orientation.\n",
      "Preprocessing rain: Flipping latitude orientation.\n",
      "Assuming correct orientation for pco2_sw. Variable has not been flipped.\n",
      "Preprocessing pco2_sw: Flipping longitude orientation.\n",
      "Assuming correct orientation for vco2_air. Variable has not been flipped.\n",
      "Preprocessing vco2_air: Flipping longitude orientation.\n",
      "Assuming correct orientation for salinity. Variable has not been flipped.\n",
      "Converting pressure from Pa to mbar\n",
      "Assuming correct orientation for pco2_sst. Variable has not been flipped.\n",
      "Preprocessing pco2_sst: Flipping longitude orientation.\n",
      "Converting pco2_sst from kelvin to celsius.\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the saline skin model (0.100000 psu added to skin salinities)\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SST gradient handling is on, using SSTskin and SSTfnd = SSTskin + 0 for flux calculation (ignoring SSTfnd data in configuration file).\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the RAPID model (from Woolf et al., 2016)\n",
      "(rate_parameterisation.py: k_Nightingale2000.__call__) Using the Nightingale et al., 2000 (N00) k parameterisation\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SUCCESS writing file /home/rr/Files/Tasks/20190408_SummerSchool/FluxEngine/output/validate_socatv4_sst_salinity_N00/2010/06/OceanFluxGHG-month06-jun-2010-v0.nc\n",
      "Flux engine exited with exit code: 0\n",
      "01 Jun 2010 00:00:00 completed successfully.\n",
      "\n",
      "Parsing settings file at: /home/rr/Files/Tasks/20190408_SummerSchool/FluxEngine/fluxengine_src/settings.xml\n",
      "(ofluxghg_flux_calc, FluxEngine._load_lon_lat_time): Grid dimensions set to: (180, 360)\n",
      "Preprocessing rain: Flipping longitude orientation.\n",
      "Preprocessing rain: Flipping latitude orientation.\n",
      "Assuming correct orientation for pco2_sw. Variable has not been flipped.\n",
      "Preprocessing pco2_sw: Flipping longitude orientation.\n",
      "Assuming correct orientation for vco2_air. Variable has not been flipped.\n",
      "Preprocessing vco2_air: Flipping longitude orientation.\n",
      "Assuming correct orientation for salinity. Variable has not been flipped.\n",
      "Converting pressure from Pa to mbar\n",
      "Assuming correct orientation for pco2_sst. Variable has not been flipped.\n",
      "Preprocessing pco2_sst: Flipping longitude orientation.\n",
      "Converting pco2_sst from kelvin to celsius.\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the saline skin model (0.100000 psu added to skin salinities)\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SST gradient handling is on, using SSTskin and SSTfnd = SSTskin + 0 for flux calculation (ignoring SSTfnd data in configuration file).\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the RAPID model (from Woolf et al., 2016)\n",
      "(rate_parameterisation.py: k_Nightingale2000.__call__) Using the Nightingale et al., 2000 (N00) k parameterisation\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SUCCESS writing file /home/rr/Files/Tasks/20190408_SummerSchool/FluxEngine/output/validate_socatv4_sst_salinity_N00/2010/07/OceanFluxGHG-month07-jul-2010-v0.nc\n",
      "Flux engine exited with exit code: 0\n",
      "01 Jul 2010 00:00:00 completed successfully.\n",
      "\n",
      "Parsing settings file at: /home/rr/Files/Tasks/20190408_SummerSchool/FluxEngine/fluxengine_src/settings.xml\n",
      "(ofluxghg_flux_calc, FluxEngine._load_lon_lat_time): Grid dimensions set to: (180, 360)\n",
      "Preprocessing rain: Flipping longitude orientation.\n",
      "Preprocessing rain: Flipping latitude orientation.\n",
      "Assuming correct orientation for pco2_sw. Variable has not been flipped.\n",
      "Preprocessing pco2_sw: Flipping longitude orientation.\n",
      "Assuming correct orientation for vco2_air. Variable has not been flipped.\n",
      "Preprocessing vco2_air: Flipping longitude orientation.\n",
      "Assuming correct orientation for salinity. Variable has not been flipped.\n",
      "Converting pressure from Pa to mbar\n",
      "Assuming correct orientation for pco2_sst. Variable has not been flipped.\n",
      "Preprocessing pco2_sst: Flipping longitude orientation.\n",
      "Converting pco2_sst from kelvin to celsius.\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the saline skin model (0.100000 psu added to skin salinities)\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SST gradient handling is on, using SSTskin and SSTfnd = SSTskin + 0 for flux calculation (ignoring SSTfnd data in configuration file).\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the RAPID model (from Woolf et al., 2016)\n",
      "(rate_parameterisation.py: k_Nightingale2000.__call__) Using the Nightingale et al., 2000 (N00) k parameterisation\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SUCCESS writing file /home/rr/Files/Tasks/20190408_SummerSchool/FluxEngine/output/validate_socatv4_sst_salinity_N00/2010/08/OceanFluxGHG-month08-aug-2010-v0.nc\n",
      "Flux engine exited with exit code: 0\n",
      "01 Aug 2010 00:00:00 completed successfully.\n",
      "\n",
      "Parsing settings file at: /home/rr/Files/Tasks/20190408_SummerSchool/FluxEngine/fluxengine_src/settings.xml\n",
      "(ofluxghg_flux_calc, FluxEngine._load_lon_lat_time): Grid dimensions set to: (180, 360)\n",
      "Preprocessing rain: Flipping longitude orientation.\n",
      "Preprocessing rain: Flipping latitude orientation.\n",
      "Assuming correct orientation for pco2_sw. Variable has not been flipped.\n",
      "Preprocessing pco2_sw: Flipping longitude orientation.\n",
      "Assuming correct orientation for vco2_air. Variable has not been flipped.\n",
      "Preprocessing vco2_air: Flipping longitude orientation.\n",
      "Assuming correct orientation for salinity. Variable has not been flipped.\n",
      "Converting pressure from Pa to mbar\n",
      "Assuming correct orientation for pco2_sst. Variable has not been flipped.\n",
      "Preprocessing pco2_sst: Flipping longitude orientation.\n",
      "Converting pco2_sst from kelvin to celsius.\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the saline skin model (0.100000 psu added to skin salinities)\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SST gradient handling is on, using SSTskin and SSTfnd = SSTskin + 0 for flux calculation (ignoring SSTfnd data in configuration file).\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the RAPID model (from Woolf et al., 2016)\n",
      "(rate_parameterisation.py: k_Nightingale2000.__call__) Using the Nightingale et al., 2000 (N00) k parameterisation\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SUCCESS writing file /home/rr/Files/Tasks/20190408_SummerSchool/FluxEngine/output/validate_socatv4_sst_salinity_N00/2010/09/OceanFluxGHG-month09-sep-2010-v0.nc\n",
      "Flux engine exited with exit code: 0\n",
      "01 Sep 2010 00:00:00 completed successfully.\n",
      "\n",
      "Parsing settings file at: /home/rr/Files/Tasks/20190408_SummerSchool/FluxEngine/fluxengine_src/settings.xml\n",
      "(ofluxghg_flux_calc, FluxEngine._load_lon_lat_time): Grid dimensions set to: (180, 360)\n",
      "Preprocessing rain: Flipping longitude orientation.\n",
      "Preprocessing rain: Flipping latitude orientation.\n",
      "Assuming correct orientation for pco2_sw. Variable has not been flipped.\n",
      "Preprocessing pco2_sw: Flipping longitude orientation.\n",
      "Assuming correct orientation for vco2_air. Variable has not been flipped.\n",
      "Preprocessing vco2_air: Flipping longitude orientation.\n",
      "Assuming correct orientation for salinity. Variable has not been flipped.\n",
      "Converting pressure from Pa to mbar\n",
      "Assuming correct orientation for pco2_sst. Variable has not been flipped.\n",
      "Preprocessing pco2_sst: Flipping longitude orientation.\n",
      "Converting pco2_sst from kelvin to celsius.\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the saline skin model (0.100000 psu added to skin salinities)\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SST gradient handling is on, using SSTskin and SSTfnd = SSTskin + 0 for flux calculation (ignoring SSTfnd data in configuration file).\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the RAPID model (from Woolf et al., 2016)\n",
      "(rate_parameterisation.py: k_Nightingale2000.__call__) Using the Nightingale et al., 2000 (N00) k parameterisation\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SUCCESS writing file /home/rr/Files/Tasks/20190408_SummerSchool/FluxEngine/output/validate_socatv4_sst_salinity_N00/2010/10/OceanFluxGHG-month10-oct-2010-v0.nc\n",
      "Flux engine exited with exit code: 0\n",
      "01 Oct 2010 00:00:00 completed successfully.\n",
      "\n",
      "Parsing settings file at: /home/rr/Files/Tasks/20190408_SummerSchool/FluxEngine/fluxengine_src/settings.xml\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(ofluxghg_flux_calc, FluxEngine._load_lon_lat_time): Grid dimensions set to: (180, 360)\n",
      "Preprocessing rain: Flipping longitude orientation.\n",
      "Preprocessing rain: Flipping latitude orientation.\n",
      "Assuming correct orientation for pco2_sw. Variable has not been flipped.\n",
      "Preprocessing pco2_sw: Flipping longitude orientation.\n",
      "Assuming correct orientation for vco2_air. Variable has not been flipped.\n",
      "Preprocessing vco2_air: Flipping longitude orientation.\n",
      "Assuming correct orientation for salinity. Variable has not been flipped.\n",
      "Converting pressure from Pa to mbar\n",
      "Assuming correct orientation for pco2_sst. Variable has not been flipped.\n",
      "Preprocessing pco2_sst: Flipping longitude orientation.\n",
      "Converting pco2_sst from kelvin to celsius.\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the saline skin model (0.100000 psu added to skin salinities)\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SST gradient handling is on, using SSTskin and SSTfnd = SSTskin + 0 for flux calculation (ignoring SSTfnd data in configuration file).\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the RAPID model (from Woolf et al., 2016)\n",
      "(rate_parameterisation.py: k_Nightingale2000.__call__) Using the Nightingale et al., 2000 (N00) k parameterisation\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SUCCESS writing file /home/rr/Files/Tasks/20190408_SummerSchool/FluxEngine/output/validate_socatv4_sst_salinity_N00/2010/11/OceanFluxGHG-month11-nov-2010-v0.nc\n",
      "Flux engine exited with exit code: 0\n",
      "01 Nov 2010 00:00:00 completed successfully.\n",
      "\n",
      "Parsing settings file at: /home/rr/Files/Tasks/20190408_SummerSchool/FluxEngine/fluxengine_src/settings.xml\n",
      "(ofluxghg_flux_calc, FluxEngine._load_lon_lat_time): Grid dimensions set to: (180, 360)\n",
      "Preprocessing rain: Flipping longitude orientation.\n",
      "Preprocessing rain: Flipping latitude orientation.\n",
      "Assuming correct orientation for pco2_sw. Variable has not been flipped.\n",
      "Preprocessing pco2_sw: Flipping longitude orientation.\n",
      "Assuming correct orientation for vco2_air. Variable has not been flipped.\n",
      "Preprocessing vco2_air: Flipping longitude orientation.\n",
      "Assuming correct orientation for salinity. Variable has not been flipped.\n",
      "Converting pressure from Pa to mbar\n",
      "Assuming correct orientation for pco2_sst. Variable has not been flipped.\n",
      "Preprocessing pco2_sst: Flipping longitude orientation.\n",
      "Converting pco2_sst from kelvin to celsius.\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the saline skin model (0.100000 psu added to skin salinities)\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SST gradient handling is on, using SSTskin and SSTfnd = SSTskin + 0 for flux calculation (ignoring SSTfnd data in configuration file).\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the RAPID model (from Woolf et al., 2016)\n",
      "(rate_parameterisation.py: k_Nightingale2000.__call__) Using the Nightingale et al., 2000 (N00) k parameterisation\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SUCCESS writing file /home/rr/Files/Tasks/20190408_SummerSchool/FluxEngine/output/validate_socatv4_sst_salinity_N00/2010/12/OceanFluxGHG-month12-dec-2010-v0.nc\n",
      "Flux engine exited with exit code: 0\n",
      "01 Dec 2010 00:00:00 completed successfully.\n",
      "\n",
      "\n",
      "\n",
      "Now calculating flux budgets...\n",
      "Using default region 'global'\n",
      "global output file  output/validate_socatv4_sst_salinity_N00/_global.txt\n",
      "No grid areas given - assuming  a1deg x 1deg grid and calculating net flux accordingly.\n",
      "Global area:  312791168.5164916 40485650.59069664 353276819.1071882\n",
      "Mean flux:  [-0.02362652  0.68184142  0.6582149 ]\n",
      "Mean flux:  [-0.0231735   0.67003199  0.64685848]\n",
      "Mean flux:  [-0.02189191  0.69917734  0.67728544]\n",
      "Mean flux:  [-0.02514122  0.69790506  0.67276385]\n",
      "Mean flux:  [-0.0276793   0.70333958  0.67566028]\n",
      "Mean flux:  [-0.02372658  0.71125025  0.68752366]\n",
      "Mean flux:  [-0.03345493  0.72388047  0.69042553]\n",
      "Mean flux:  [-0.02607147  0.69510908  0.66903761]\n",
      "Mean flux:  [-0.0183112   0.69194954  0.67363835]\n",
      "Mean flux:  [-0.02576866  0.6898664   0.66409774]\n",
      "Mean flux:  [-0.03224291  0.66507496  0.63283206]\n",
      "Mean flux:  [-0.02629014  0.65859446  0.63230432]\n",
      "Yearly mean flux (2010): [-0.02558714  0.69051561  0.66492847]\n",
      "\n",
      "\n",
      "Comparing output to reference data...\n",
      "\n",
      "Validation successful! All values are within threshold limits:\n",
      "\tMISSING GROSS DOWNWARD FLUX TgC: 100.0%\n",
      "\t30.5 DAY GROSS UPWARD FLUX TgC: 100.0%\n",
      "\tMISSING FLUX TgC: 100.0%\n",
      "\tGROSS DOWNWARD FLUX TgC: 100.0%\n",
      "\tNET FLUX TgC: 100.0%\n",
      "\t30.5 DAY GROSS DOWNWARD FLUX TgC: 100.0%\n",
      "\tMISSING GROSS UPWARD FLUX TgC: 100.0%\n",
      "\t30.5 DAY FLUX TgC: 100.0%\n",
      "\tGROSS UPWARD FLUX TgC: 100.0%\n"
     ]
    }
   ],
   "source": [
    "#Run the verification script.\n",
    "!python verify_socatv4_sst_salinity_gradients_N00.py #Run the verification script"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Creating a custom configuration file\n",
    "For the remainder of this tutorial we will create a new run configuration, based on the verification run but with a number of changes, and then visualise the output. To do this, you'll first need to understand FluxEngine configuration files.\n",
    "\n",
    "Lets have a look at the configuration file used to run the above verification. FluxEngine uses plain text configuration files to specify input data and to select various options to change how the flux calculation is performed. We'll be using this configuration as the basis for our new FluxEngine run, but it's important not to modify the original or future verification runs may not work. To make a copy of the file run the following:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os.path as path;\n",
    "import shutil;\n",
    "shutil.copy(path.join(\"configs\", \"socatv4_sst_salinity_gradients-N00.conf\"), path.join(\"Tutorials\", \"01_introduction\", \"custom_config.conf\"));"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understanding configuration files\n",
    "We now have made a new configuration file based on the verification run, lets open it. You can find it here `FluxEngine/Tutorials/01_introduction/custom_config.conf`. Use a text editor such as Notepad++ to open this file and view its contents.\n",
    "\n",
    "<div class=\"alert alert-block alert-warning\">\n",
    "<b>Note - Opening config files: </b> Configuration files are in plain text format and can be opened in software such as Notepad, Notepad++ or TextEdit. You should not use Microsoft Word or other word processing software to edit configuration files because they can add invisible formatting characters to files which can prevent the FluxEngine interpretting the file correctly. On some older Windows systems configuration files may display all on one line. The easiest way to avoid this is by installing Notepad++ (a free lightweight text editor), and using this to edit configuration files.\n",
    "</div>\n",
    "\n",
    "Configuration files contain a list of options/setting names and values following the format of `option = value`. The order in which options are defined does not matter, but it is often useful to group related options together. Any text preceeded by a `#` symbol is a comment. Comments provide helpful information about what a particular setting, or group of settings, do. It isn't necessary to understand all of the settings at this point, but see if you can identify where the following settings are defined by reading the comments:\n",
    " - Where is input data specified? What types of input data are supplied (e.g. salinity data)?\n",
    " - Where is the flux calculation selected? Which flux equation (e.g. the 'rapid' or 'bulk') does the verification run use?\n",
    " - Where is gas transfer velocity parameterisation specified?\n",
    " - Where is the output directory is set? Optional: Locate the output directory and open some one of the files (see the information box below for help).\n",
    "\n",
    "If you want to find out more information about a particular setting you can look the description in [the instructions](https://github.com/oceanflux-ghg/FluxEngine/blob/master/FluxEngineV3_instructions.pdf), (see section 7.2).\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Extra info - Opening netCDF files:</b> The easiest way to open netCDF files is to use a program such as Panoply. You can download Panoply for Mac, Windows or Linux [here](https://www.giss.nasa.gov/tools/panoply/). Alternatively, you can use a programming language such as Python, R or Matlab to read netCDF files and plot them. This is beyond the scope of these tutorials, and it is recommended that you use Panoply for now, unless you already have experience reading and plotting netCDF files.\n",
    "</div> \n",
    "\n",
    "## Modifying the config file\n",
    "\n",
    "### Specifying input data\n",
    "Each input dataset is described as a data layer, within FluxEngine. These are supplied as netCDF (.nc) files which contain metadata about the data (description, units, expected range, etc.) as well as data about the dimensions (spatial and temporal coordinates). An input data layer is defined by a file path and a 'product' name:\n",
    "\n",
    "```\n",
    "windu10_path = data/validation_data/globwave/<YYYY>/<YYYY><MM>_OCF-WSP-GLO-1M-100-MGD-GW-v2.nc\n",
    "windu10_prod = wind_speed_cor_mean\n",
    "```\n",
    "\n",
    "Here we're defining the file path to our wind speed netCDF file using `windu10_path`, and the product name (which is the name of the variable inside the netCDF file) using `windu10_prod`. Notice that tokens are used to represent the year and month (`<YYYY>` and `<MM>`). These are substituded for the numerical representation of the current year and month when FluxEngine runs, allowing different input files to be selected for different time steps.\n",
    "\n",
    "We will use an updated salinity dataset from the World Ocean Atlas (WOA) for our custom flux calculation. Find the section of the configuration file which defines the salinity input and modify them to look like this:\n",
    "\n",
    "```\n",
    "salinity_path = data/tutorial_data/WOA_salinity/surface_woa18_A5B7_s<MM>_01.nc\n",
    "salinity_prod = salinity_mean\n",
    "```\n",
    "\n",
    "Notice that if you open one of the WOA salinity data netCDF files using Panoply there are three variables, `lon`, `lat` and `mean_salinity`. The first two define the lon-lat grid and the last contains the mean monthly sea surface salinity. We want the sea surface salinity, which is why we set `salinity_prod = salinity_mean`.\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Extra info:</b> There is a general pattern to specifying data layers which is `datalayername_suffix`. You've already seen the use of the `_path` and `_prod` suffix to define file paths and product names, but others can be used to define, for example, minimum and maximum allowed bounds for the input data, or to perform preprocessing such as unit conversions. [The instructions](https://github.com/oceanflux-ghg/FluxEngine/blob/master/FluxEngineV3_instructions.pdf) (section 7.6.1) provides more information on using different suffixes to change the way input data layers are handled.\n",
    "</div>\n",
    "\n",
    "\n",
    "### Selecting a gas transfer velocity parameterisation\n",
    "We'll use a more recent wind-based gas transfer velocity parameterisation too. Find the line that starts `k_parameterisation`. It's currently set to use the parameterisation described in Nightingale et al., Global Biogeochemical Cycles, 2000. There are a number of different gas transfer velocity parameterisations built in to FluxEngine, and users can add custom one. We'll stick to using a build-in parameterisation for now (many of these are listed in the comments of the configuration file. Set `k_parameterisation = k_Wanninkhof2014`, which tell FluxEngine to use the updated parameterisation described by Wanninkhof, Limnology and Oceanography: Methods, 2014.\n",
    "\n",
    "We'll also update the Schmidt number parameterisation to use the relationship described in the same paper. Currently the configuration file doesn't define a Schmidt number parameterisation and so it will use the default version which is outdated. Add a new line to the configuration file that reads `schmidt_parameterisation = schmidt_Wanninkhof2014`. Now our gas transfer parameterisation and Schmidt number calculation will be consistent. Note that it doesn't matter where you add this line, but it might help you find it again if you add it near the k parameterisation line.\n",
    "\n",
    "### Setting the output directory\n",
    "Finally we need to change the output directory, otherwise we will overwrite the output produced by the verification run. Set the `output_dir` setting in the configuration file to `output/tutorial_output_introduction`.\n",
    "\n",
    "By default, FluxEngine will sort output files into subdirectories based on year and month. We're only going to run our new configuration file for a single month, so we can overwrite the output directory structure by adding the following line:\n",
    "\n",
    "`output_structure =`\n",
    "\n",
    "Note that we are purposefully leaving the value empty, telling FluxEngine to use the root output directory (i.e. the one defined in `output_dir`).\n",
    "\n",
    "Make sure you save the configuration file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running FluxEngine with our custom configuration\n",
    "Now all that's left to do is to run FluxEngine with out new configuration file. There are two ways to run FluxEngine. The first is using the commandline tool and specifying a configuration file, a start date and a stop date. The second is to import FluxEngine as a Python module and write a custom script to drive it. We'll be using the simpler commandline tool approach. The commandline tool can do nearly everything the Python module can. The main advantage of using the Python module is that it allows you to run many similar flux calculations with slight changes, for example if you want to run a sensitivity analysis.\n",
    "\n",
    "We'll use the commandline tool (which is written in Python) by running `python ofluxghg_run.py \"Tutorials/01_introduction/custom_config.conf\" -s \"2010-01-01\" -e \"2010-01-31\"` in your Command Prompt (windows) or Terminal (Mac/Linux) window. The first part of the command `python ofluxghg_run.py` tell your computer to use Python to run the commandline tool which will in turn setup and run FluxEngine. The next part of the command specifies the path to the configuration file to use. Next the `-s` and `-e` specify the start and end dates, which are provided in the `YYYY-MM-DD`. For more information on using the `ofluxghg_run.py` tool you can run it with the `-h` comment ('h' for help) or see [the instructions](https://github.com/oceanflux-ghg/FluxEngine/blob/master/FluxEngineV3_instructions.pdf) (section 5.1).\n",
    "\n",
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Jupyter notebooks and the commandline:</b> Code which starts with `!` is interpretted as a commandline command, and is equivalent to entering the command (without the `!`) into your command prompt (Windows) or terminal (Mac/Linux) window.\n",
    "</div>\n",
    "\n",
    "Run the command below to perform our new flux calculation. Notice, to minimise execution time we're only calculating the fluxes for January."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Executing on 'rr-HP-Compaq-Elite-8300-MT' at 21/05/2019 14:11:53\n",
      "Parsing settings file at: /home/rr/Files/fluxengine_v3/FluxEngine/fluxengine_src/settings.xml\n",
      "(ofluxghg_flux_calc, FluxEngine._load_lon_lat_time): Grid dimensions set to: (180, 360)\n",
      "Assuming correct orientation for pco2_sw. Variable has not been flipped.\n",
      "Preprocessing pco2_sw: Flipping longitude orientation.\n",
      "Assuming correct orientation for vco2_air. Variable has not been flipped.\n",
      "Preprocessing vco2_air: Flipping longitude orientation.\n",
      "Converting pressure from Pa to mbar\n",
      "Assuming correct orientation for pco2_sst. Variable has not been flipped.\n",
      "Preprocessing pco2_sst: Flipping longitude orientation.\n",
      "Converting pco2_sst from kelvin to celsius.\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the saline skin model (0.100000 psu added to skin salinities)\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SST gradient handling is on, using SSTskin and SSTfnd = SSTskin + 0 for flux calculation (ignoring SSTfnd data in configuration file).\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) Using the RAPID model (from Woolf et al., 2016)\n",
      "(rate_parameterisation.py: k_Wanninkhof2014.__call__) Using the Wanninkhof 2014 k parameterisation\n",
      "(ofluxghg_flux_calc, FluxEngine._run_fluxengine_) SUCCESS writing file /home/rr/Files/fluxengine_v3/FluxEngine/output/tutorial_output_introduction/OceanFluxGHG-month01-jan-2010-v0.nc\n",
      "Flux engine exited with exit code: 0\n",
      "01 Jan 2010 00:00:00 completed successfully.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!python ofluxghg_run.py \"Tutorials/01_introduction/custom_config.conf\" -s \"2010-01-01\" -e \"2010-01-31\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualising output\n",
    "If all has gone well you'll see a log of the FluxEngine run and a message at the end saying 'completed successfully'. If you look at the output directory `output/tutorial_output_introduction/` you should see a file called `OceanFluxGHG-month01-jan-2010-v0.nc`. If you have Panoply installed, you can open this file directly and look at its contents."
   ]
  },
  {
   "attachments": {
    "Panoply.png": {
     "image/png": "[encoded data, 182496 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Panoply.png](attachment:Panoply.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you double click a variable name (e.g. 'OF' - the air-sea gas flux) Panoply will give you the option to plot it. Spend some time exploring the outputs FluxEngine provides. Outputs that may be of interest are the air-sea gas flux (OF), the gas transfer velocity (OK3) and the interface and aqueous gas concentrations (OIC1 and OSFC, respectively). You'll also notice that a copy of the input data sets are provided in the output file. This makes it convenient to see what values were used to make the calculation.\n",
    "\n",
    "Plotting in Panoply is fine, but it is not very flexible (for example, what if we want to compare the calculated CO<sub>2</sub> fluxes between the verification run and our custom run). You can export data to a .csv file by right clicking on a variable and choosing one of the export options. Then you can import the data into your preferred data visualisation / analysis software and perform any operations you like. However, since we're already running a Python interpretter in this Jupyter Notebook, we might as well do this directly with Python! Lets compare the air-sea CO<sub>2</sub> fluxes between the verification and custom runs now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 84240 characters]\n",
      "text/plain": [
       "<Figure size 576x576 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from netCDF4 import Dataset #allows reading of netCDF files\n",
    "import matplotlib.pyplot as plt #for plotting data\n",
    "import numpy as np #matrix manipulation\n",
    "\n",
    "verificationNetCDF = Dataset(\"output/validate_socatv4_sst_salinity_N00/2010/01/OceanFluxGHG-month01-jan-2010-v0.nc\", 'r') #Read the January data from the verification run\n",
    "verificationFlux = verificationNetCDF.variables[\"OF\"][:] #Extract the flux data from the 'OF' variable in the netCDF file\n",
    "\n",
    "customNetCDF = Dataset(\"output/tutorial_output_introduction/OceanFluxGHG-month01-jan-2010-v0.nc\", 'r') #Read the January data from the custom run\n",
    "customFlux = customNetCDF.variables[\"OF\"][:] #Extract the flux data from the 'OF' variable in the netCDF file\n",
    "\n",
    "fluxDifference = customFlux - verificationFlux #Calculate the change in calculated flux\n",
    "fluxDifference = np.squeeze(fluxDifference) #remove any dimensions of size 1 (in this case, the time dimension)\n",
    "\n",
    "#plot the differences in flux\n",
    "plt.figure(figsize=(8, 8))\n",
    "plt.imshow(fluxDifference)\n",
    "cbar = plt.colorbar(orientation=\"horizontal\")\n",
    "cbar.set_label(r\"Difference in flux (gC m$^{-2}$ s$^{-1}$)\", fontsize=16)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The difference biggest differences in calculated fluxes appear around regions with high wind speed (exceeding around 15 ms<sup>-1</sup>). The change in salinity data is unlikely to have a big effect, and so it seems likely that these differences are due to the updated gas transfer velocity parameterisation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Next tutorial\n",
    "Next, we'll look at how to use FluxEngine to calculate gas fluxes using in situ data. Return to the Jupyter dashboard and open the notebook stored in [Tutorials/02_using_insitu_data/using_insitu_data.ipynb](../../Tutorials/02_using_insitu_data/using_insitu_data.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.15"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
