{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Calculating air-sea fluxes of nitrous oxide (N<sub>2</sub>O)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "FluxEngine provides support for calculating atmosphere-ocean gas fluxes of nitrous oxide (N<sub>2</sub>O) and methane. These are both powerful, but poorly soluble, greenhouse gases produced by ocean ecosystems. The ocean is an important source of N<sub>2</sub>O and in this tutorial we will use FluxEngine to calculate air-sea N<sub>2</sub>O fluxes using data collected from an [AMT cruise](http://www.amt-uk.org/) (and stored in the [MEMENTO](https://memento.geomar.de/) database).\n",
    "\n",
    "In addition to simply calculating the air-sea flux of N<sub>2</sub>O, we'll use FluxEngine to assess the impact of biological surfactants on air-sea exchange (more on that later)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Set the working directory\n",
    "The first thing we need to do is create a new directory for the files we'll create during this tutorial and set it as our working directory:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#This code changes the working directory from the tutorial folder in your FluxEngine installation\n",
    "# to a newly created folder in your home drive\n",
    "import os;\n",
    "import matplotlib.pyplot as plt; #Used later for plotting\n",
    "\n",
    "homeDirectory = os.path.expanduser(\"~\");\n",
    "workingDirectory = os.path.join(homeDirectory, \"FluxEngineTutorials\", \"tutorial_04\"); #Change this line to change th default location for files you create in this tutorial.\n",
    "\n",
    "if os.path.exists(workingDirectory) == False: #If the new directory doesn't already exist, create it\n",
    "    os.makedirs(workingDirectory);\n",
    "os.chdir(workingDirectory);\n",
    "print(\"Working directory is now:\", os.getcwd());"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## These data\n",
    "We'll be using with data from the [AMT-24](http://www.amt-uk.org/Cruises/AMT24) cruise which took place in 2014. N<sub>2</sub>O data has been provided for the October stretch of the cruise. so we just need to copy it to our working directory. Later in this tutorial we will also need some wind speed and air pressure data. These have also been provided, so we'll copy them all to a subdirectory called `copied_data` in our working directory. To do this run the below code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from fluxengine.core.fe_setup_tools import get_fluxengine_root; #Gets filepath to fluxengine root directory\n",
    "from os import mkdir, path, getcwd; #cross-platform filepath manipulation and access to current working directly\n",
    "import shutil;\n",
    "\n",
    "#Copy Reynolds OISST data\n",
    "try:\n",
    "    copyFrom = path.join(get_fluxengine_root(), \"tutorials\", \"04_N2O_fluxes\", \"data\");\n",
    "    copyTo = path.join(getcwd(), \"copied_data\");\n",
    "    shutil.copytree(copyFrom, copyTo);\n",
    "    print(\"Tutorial data (including MEMENTO N2O data) have been successfully copied to:\", copyTo);\n",
    "except Exception as e:\n",
    "    print(\"There was a problem copying the tutorial data. The folder\", copyTo, \"may already exist. If so, try deleting it before rerunning this cell.\");\n",
    "    print(e);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The N2O data that we just copied are already in netCDF format and can be found in the `copied_data/MEMENTO_N2O` directory. Open the file in Panoply to see and/or plot the variables.\n",
    "\n",
    "<img src=\"images/AMT24-datapng.png\">\n",
    "\n",
    "<!-- ![AMT24-datapng.png](attachment:AMT24-datapng.png) -->\n",
    "\n",
    "The data file contains nearly all the information we need to calculate gas fluxes. Included are:\n",
    "- atmospheric gas partial pressure (`partial_pressure_atm`)\n",
    "- aqueous gas partial pressure (`partial_pressure_ocean`)\n",
    "- salinity (`Sal`)\n",
    "- sub-skin sea surface temperature (`SST`)\n",
    "- longitude (`longitude`), latitude(`latitude`), and time (`month`) dimension information\n",
    "\n",
    "All we are missing is wind speed data and air pressure data. We will use [Cross-Calibrated Multi-Platform (CCMP) wind speed data](http://www.remss.com/measurements/ccmp/) and [European Centre for Medium-Range Weather Forecasts (ECMWF) monthly composite air pressure](https://www.ecmwf.int/en/forecasts/datasets/archive-datasets) data to complement the in situ measurements. For convenience these have already supplied in netCDF format (and we've already copied them to the `copied_data` directory of the working directory.\n",
    "\n",
    "Note that the reanalysis tool is only suitable for using with CO<sub>2</sub> data, so we won't be using it here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating the configuration file\n",
    "Let's start by copying the configuration file we used for the previous AMT cruise tutorial, and using that as a template to modify. You could use your file browser for this, but it's easiest to just run the code below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fluxengine.core.fe_setup_tools import get_fluxengine_root;\n",
    "import os.path as path;\n",
    "import shutil;\n",
    "copyFrom = path.join(get_fluxengine_root(), \"tutorials\", \"02_using_insitu_data\", \"kitidis_cruise_W14_reanalysed.conf\");\n",
    "copyTo = path.join(\"N2O_rapid_N00.conf\"); #relative to the working directory\n",
    "shutil.copy(copyFrom, copyTo);\n",
    "print(\"Configuration file copied to:\", copyTo);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Input data specification\n",
    "The first thing to do is to change our input data specification. Open the newly copied configuration file (`N2O_rapid_N00.conf`) and scroll to input specification section of the configuration file. Where possible, we want to use inputs from our MEMENTO netCDF file for the cruise (`copied_data/MEMENTO_N2O/N2O_onedeggrid_monthly_2014_10.nc`). This means you should update the input data layer path and product names for the atmospheric and aqueous pN<sub>2</sub>O, SST, and salinity. Make sure you set the product names correctly (you can find these either from the bullet point list above, or by opening the file in Panoply and reading the metadata for each variable. Note that for the atmospheric and aqueous gas partial pressures the datalayer name will still have 'co2' in it. This doesn't matter as it really just stands for whichever gas we're interested in calculating the fluxes for.\n",
    "\n",
    "For air pressure we will use the ECMWF data, so specify the path as `copied_data/201410_OCF-PRE-GLO-1M-100-ECMWF.nc`. Using Panoply to open this file you'll see that the correct product name for mean sea level air pressure is `msl_mean`.\n",
    "\n",
    "For wind speed we want the CCMP data so specify the path as `copied_data/ccmp_v2_rebinned_2014_month10.nc` and use `windspeed` for the product name. If you open this file in Panoply you'll see that there are no variables for the second and third moment of wind speed. Not to worry - we can use FluxEngine's preprocessing feature to approximate these using the second and third power of wind speed. To do this use the same path and product name as you used for the wind speed, but add the following preprocessing functions:\n",
    "```\n",
    "windu10_moment2_preprocessing = pow2\n",
    "windu10_moment3_preprocessing = pow3\n",
    "```\n",
    "\n",
    "Some of our input data are not in the correct units for FluxEngine to work with. Section 6 of [the manual](https://github.com/oceanflux-ghg/FluxEngine/blob/master/FluxEngineV4_instructions.pdf) tells us that air pressure should be supplied in millibars, but the ECMWF data are in pascals. We can use a preprocessing function to tell FluxEngine to convert this into the correct units before using it in the calculation. At the bottom of the air pressure input specification, add a new line:\n",
    "`pressure_preprocessing = pascal_to_millibar`\n",
    "\n",
    "We need to make one final change to the input data specification. You may have noticed that the netCDF file containing the MEMENTO data provided the atmospheric N<sub>2</sub>O values as partial pressure rather than a molar fraction. This isn't a problem, because FluxEngine will convert between the two, but we need to tell FluxEngine which input we're providing. To do this we should change `vgas_air_path` and `vgas_air_prod` to `pgas_air_path` and `pgas_air_prod`. FluxEngine also expects gas partial pressure data to be in microatm but the MEMENTO data we have uses nanoatm. We'll use another preprocessing function to make this conversion:\n",
    "```\n",
    "pgas_air_preprocessing = nano_to_micro\n",
    "pgas_sw_preprocessing = nano_to_micro\n",
    "```\n",
    "\n",
    "Once you're finished, the input specification section of the configuration file should look like this:\n",
    "\n",
    "```\n",
    "# sst skin data inputs\n",
    "sstfnd_path = copied_data/MEMENTO_N2O/N2O_onedeggrid_monthly_2014_10.nc\n",
    "sstfnd_prod = SST\n",
    "sstfnd_preprocessing = celsius_to_kelvin\n",
    "sstfnd_timeDimensionName = month\n",
    "\n",
    "\n",
    "# U10 wind data inputs\n",
    "windu10_path = copied_data/ccmp_v2_rebinned_2014_month10.nc\n",
    "windu10_prod = windspeed\n",
    "\n",
    "windu10_moment2_path = Tutorials/04_N2O_fluxes/data/ccmp_v2_rebinned_2014_month10.nc\n",
    "windu10_moment2_prod = windspeed\n",
    "windu10_moment2_preprocessing = pow2\n",
    "\n",
    "windu10_moment3_path = Tutorials/04_N2O_fluxes/data/ccmp_v2_rebinned_2014_month10.nc\n",
    "windu10_moment3_prod = windspeed\n",
    "windu10_moment3_preprocessing = pow3\n",
    "\n",
    "\n",
    "# modelled air pressure data.\n",
    "pressure_path = Tutorials/04_N2O_fluxes/data/201410_OCF-PRE-GLO-1M-100-ECMWF.nc\n",
    "pressure_prod = msl_mean\n",
    "pressure_preprocessing = pascal_to_millibar\n",
    "\n",
    "\n",
    "# Salinity data\n",
    "salinity_path = Tutorials/04_N2O_fluxes/data/MEMENTO_N2O/N2O_onedeggrid_monthly_2014_10.nc\n",
    "salinity_prod = Sal\n",
    "salinity_timeDimensionName = month\n",
    "\n",
    "\n",
    "#atmospheric and ocean CO2 data\n",
    "pgas_air_path = Tutorials/04_N2O_fluxes/data/MEMENTO_N2O/N2O_onedeggrid_monthly_2014_10.nc\n",
    "pgas_air_prod = partial_pressure_atm\n",
    "pgas_air_timeDimensionName = month\n",
    "pgas_air_preprocessing = nano_to_micro\n",
    "\n",
    "pgas_sw_path = Tutorials/04_N2O_fluxes/data/MEMENTO_N2O/N2O_onedeggrid_monthly_2014_10.nc\n",
    "pgas_sw_prod = partial_pressure_ocean\n",
    "pgas_sw_timeDimensionName = month\n",
    "pgas_sw_preprocessing = nano_to_micro\n",
    "```\n",
    "\n",
    "### Configuring the dimensions\n",
    "We need to tell FluxEngine which input field to extract the dimension data from (longitude, latitude and temporal dimensions). Scrolling to near the top of the configuration file you'll see that the `axes_data_layer` option is currently set to `sstfnd`, which is extracted from our MEMENTO AMT cruise data file. If you open this file with Panoply (`copied_data/MEMENTO_N2O/N2O_onedeggrid_monthly_2014_10.nc`) you'll see that the spatial dimensions are named `longitude` and `latitude`. This is ideal, because those are the names already set for `longitude_prod` and `latitude_prod`.\n",
    "\n",
    "Similarly the `time_prod` option is set to `time`, which is what we need for the CCMP and ECMWF data. Unfortunately, not all of our input data uses `time` as their temporal dimension name. The MEMENTO AMT netCDF file uses `month` instead. We can override the default `time` for these inputs by adding the following lines (ideally, just after the product name for the corresponding input field):\n",
    "```\n",
    "sstfnd_timeDimensionName = month\n",
    "salinity_timeDimensionName = month\n",
    "pgas_air_timeDimensionName = month\n",
    "pgas_sw_timeDimensionName = month\n",
    "```\n",
    "Now, when FluxEngine reads in the data from these files, it will know to look for `month` and not the default `time` to determine the dimensions of these fields.\n",
    "\n",
    "\n",
    "### Gas settings and gas transfer velocity parameterisation\n",
    "Next, we need to tell FluxEngine to use the [Nightingale 2000](https://agupubs.onlinelibrary.wiley.com/doi/abs/10.1029/1999GB900091) gas transfer velocity. We're using this parameterisation because there is a version available which estimates the effect of surfactants on gas transfer, and this will allow us to compare like-for-like later in this tutorial. Find the line that reads `k_parameterisation = k_Wanninkhof2014` and change it to `k_parameterisation = k_Nightingale2000`.\n",
    "\n",
    "We also need to specify the gas that we will calculate fluxes for. This is needed to correctly calculate the solubility of the gas and to correctly perform the Schmidt number calculation. Add the following lines after the `k_parameterisation` line:\n",
    "```\n",
    "GAS = N2O #calculating fluxes for nitrous oxide, not CO2\n",
    "ATMGAS = \"P\" #Atmospheric gas data is supplied as partial pressure\n",
    "```\n",
    "\n",
    "### Output file specification\n",
    "Finally, we should specify a meaningful output directory and filename. Scroll to the output path specification section of the configuration file and modify `output_dir` and `output_file` to look like this:\n",
    "```\n",
    "output_dir = output/memento_n2o_runs/\n",
    "output_file = AMT_N2O_N00.nc\n",
    "output_structure = #blank\n",
    "```\n",
    "\n",
    "That's everything we need to change! If you get stuck, a completed version of the configuration file has been included in the `tutorials` directory of wherever you installed FluxEngine to: `tutorials/04_N2O_fluxes/_complete/N2O_rapid_N00_complete.conf`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running FluxEngine\n",
    "Use the command line tool to run FluxEngine with our new configuration file. Notice that the start and stop options are set to just calculate the fluxes for October."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!fe_run.py \"N2O_rapid_N00.conf\" -s \"2014-10-01\" -e \"2014-11-01\" -l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should now have a new FluxEngine output file in `output/memento_n2o_runs/`. Feel free to open this file and visualise it in Panoply."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Estimating the effect of surfactant suppression\n",
    "Surfactants are compounds on the surface of the ocean which can suppress turbulence at the surface. These are usually naturally occurring but can have a significant impact on the exchange of gases between the atmosphere and ocean surface. It would be interesting to quantify the expected effect of surfactants on N<sub>2</sub>O fluxes for the cruise data. [Pereira et al (2018)](https://www.nature.com/articles/s41561-018-0136-2.pdf?origin=ppub) developed a parameterisation for estimating the suppression effect of surfactants from skin sea surface temperature.\n",
    "\n",
    "FluxEngine provides the ability to use custom written gas transfer velocity parameterisations using Python, so it was possible to write a custom parameterisation that implements this suppression estimate. This was done for a previous study and the parameterisation is now available in the FluxEngine download. To use it, we just need to create a new configuration file that specifies the custom gas transfer parameterisation.\n",
    "\n",
    "Make a copy of the configuration file we just used by running the code below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os.path as path\n",
    "import shutil\n",
    "\n",
    "copyFrom = path.join(\"N2O_rapid_N00.conf\");\n",
    "copyTo = path.join(\"N2O_rapid_N00_surfactants.conf\");\n",
    "shutil.copy(copyFrom, copyTo);\n",
    "print(\"Configuration file copied to:\", copyTo);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Open the new configuration file (`N2O_rapid_N00_surfactants.conf`) and find the line that specifies the gas transfer velocity (k) parameterisation. Change this from `k_parameterisation = k_Nightingale2000` to `k_parameterisation = k_Nightingale2000_with_surfactant_suppression`.\n",
    "\n",
    "We also need to change the output file name, or we'll overwrite the output we just ran. Set the `output_file` option to be `AMT_N2O_N00_surfactants.nc`.\n",
    "\n",
    "That's all you need to do! Run FluxEngine using the command line tool, just as we did before, but using the new configuration file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!fe_run.py \"N2O_rapid_N00_surfactants.conf\" -s \"2014-10-01\" -e \"2014-11-01\" -l"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Open the `output/memento_n2o_runs` directory and you'll see that we have two output files, one called `AMT_N2O_N00_surfactants.nc` (with surfactant suppression) and one called `AMT_N2O_N00.nc` (without surfactant suppression).\n",
    "\n",
    "Let's calculate the difference in gas transfer velocity and air-sea N<sub>2</sub>O flux variables with and without the surfactant suppression. You could open the output files in Panoply and export the data as a comma separated variable file, or open the netCDF files directly using a programming language such as Python, R or Matlab. We're using Python here, but it doesn't matter if you don't follow the specifics of the code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "from netCDF4 import Dataset\n",
    "import numpy as np\n",
    "\n",
    "#read the flux engine output files into memory\n",
    "withoutSurfactants = Dataset(\"output/memento_n2o_runs/AMT_N2O_N00.nc\", 'r')\n",
    "withSurfactants = Dataset(\"output/memento_n2o_runs/AMT_N2O_N00_surfactants.nc\", 'r')\n",
    "\n",
    "#calculate difference in gas transfer velocity due to surfactant suppression\n",
    "kDiff = withSurfactants.variables[\"OK3\"][0,:,:] - withoutSurfactants.variables[\"OK3\"][0,:,:]\n",
    "\n",
    "#calculate difference in gas tflux due to surfactant suppression\n",
    "fluxDiff = withSurfactants.variables[\"OF\"][0,:,:] - withoutSurfactants.variables[\"OF\"][0,:,:]\n",
    "\n",
    "#We'll plot this by latitude, so calculate the mean difference across each latitude band\n",
    "kDiff = np.nanmean(kDiff, axis=1)\n",
    "fluxDiff = np.nanmean(fluxDiff, axis=1)\n",
    "\n",
    "#remove nan values to create a continuous dataset\n",
    "toKeepK = np.where(np.isfinite(kDiff))\n",
    "toKeepFlux = np.where(np.isfinite(fluxDiff))\n",
    "\n",
    "#create plot\n",
    "f, (ax1, ax2) = plt.subplots(2, figsize=[10,6])\n",
    "\n",
    "latitude = np.arange(89.5, -90.5, -1)\n",
    "ax1.plot(latitude[toKeepK], kDiff[toKeepK])\n",
    "ax1.set_xlabel(\"latitude\")\n",
    "ax1.set_ylabel(\"difference in gas transfer velocity\\n\"+r\"(cm hr$^{-1}$)\")\n",
    "\n",
    "ax2.plot(latitude[toKeepFlux], fluxDiff[toKeepFlux])\n",
    "ax2.set_xlabel(\"latitude\")\n",
    "ax2.set_ylabel(\"difference in sea-to-air flux\\n\"+r\"(g C m$^{-2}$ day$^{-1}$)\")\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The top plot shows the difference in gas transfer velocity due to the effect of surfactants. The fact that this difference is negative reinforces the that the effect of surfactants is to suppress gas transfer. The effect on the sea-to-air N<sub>2</sub>O flux is mixed, however. This is because the flux is positive in some regions (net source to the atmosphere) and negative in other regions (net sink to the ocean). Regardless of direction, the effect of surfactants is to reduce the magnitude of the flux.\n",
    "\n",
    "This is the final tutorial. There is a lot more information available in [the manual](https://github.com/oceanflux-ghg/FluxEngine/blob/master/FluxEngineV4_instructions.pdf), including descriptions of the options available in configuration files, descriptions of different ways to include input data and information on writing custom preprocessing and gas transfer velocity functions.\n",
    "\n",
    "Questions, bugs reports and general feedback can be directed to Tom Holding at t.m.holding@exeter.ac.uk."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
