{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ourownstory/neural_prophet/blob/master/tutorials/feature-use/sparse_autoregression_yosemite_temps.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Sparse Autoregression\n",
    "Here we fit NeuralProphet to data with 5-minute resolution (daily temperatures at Yosemite). \n",
    "This is a continuation of the example notebook `autoregression_yosemite_temps`, focusing on sparsity.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "if 'google.colab' in str(get_ipython()):\n",
    "    !pip install git+https://github.com/ourownstory/neural_prophet.git # may take a while\n",
    "    #!pip install neuralprophet # much faster, but may not have the latest upgrades/bugfixes\n",
    "\n",
    "import pandas as pd\n",
    "from neuralprophet import NeuralProphet, set_log_level\n",
    "set_log_level(\"ERROR\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_location = \"https://raw.githubusercontent.com/ourownstory/neuralprophet-data/main/datasets/\"\n",
    "df = pd.read_csv(data_location + \"yosemite_temps.csv\")\n",
    "# df.head(3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sparsifying the AR coefficients\n",
    "The autoregression component of NeuralProphet is defined as a AR-Net ([paper](https://arxiv.org/abs/1911.12436), [github](https://github.com/ourownstory/AR-Net)).\n",
    "Thus, we can set `ar_sparsity` to a number smaller one, if we like to induce sparsity in the AR coefficients. \n",
    "\n",
    "However, fitting a model with multiple components and regularizations can be harder to fit and in some cases you may need to take manual control over the training hyperparameters.\n",
    "\n",
    "\n",
    "We will start by setting a sparsity to 50%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                                                                                  \r"
     ]
    }
   ],
   "source": [
    "m = NeuralProphet(\n",
    "    n_lags=6*12,\n",
    "    n_forecasts=3*12,\n",
    "    n_changepoints=0,\n",
    "    weekly_seasonality=False,\n",
    "    daily_seasonality=False,\n",
    "    learning_rate=0.01,\n",
    "    ar_sparsity=0.9,\n",
    ")\n",
    "metrics = m.fit(df, freq='5min') # validate_each_epoch=True, plot_live_loss=True"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 30604 characters]",
      "text/plain": [
       "<Figure size 720x432 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig_param = m.plot_parameters()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 30000 characters]",
      "text/plain": [
       "<Figure size 720x432 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "m = m.highlight_nth_step_ahead_of_each_forecast(1)\n",
    "fig_param = m.plot_parameters()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 29512 characters]",
      "text/plain": [
       "<Figure size 720x432 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "m = m.highlight_nth_step_ahead_of_each_forecast(36)\n",
    "fig_param = m.plot_parameters()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further reducing the non-zero AR-coefficents\n",
    "By setting the ar_sparsity lower, we can further reduce the number of non-zero weights.\n",
    "Here we set it to 10%"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                                                                                  \r"
     ]
    }
   ],
   "source": [
    "m = NeuralProphet(\n",
    "    n_lags=6*12,\n",
    "    n_forecasts=3*12,\n",
    "    n_changepoints=0,\n",
    "    daily_seasonality=False,\n",
    "    weekly_seasonality=False,\n",
    "    learning_rate=0.01,\n",
    "    ar_sparsity=0.5,\n",
    ")\n",
    "metrics = m.fit(df, freq='5min')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 30396 characters]",
      "text/plain": [
       "<Figure size 720x432 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig_param = m.plot_parameters()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 30100 characters]",
      "text/plain": [
       "<Figure size 720x432 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "m = m.highlight_nth_step_ahead_of_each_forecast(1)\n",
    "fig_param = m.plot_parameters()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 30024 characters]",
      "text/plain": [
       "<Figure size 720x432 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "m = m.highlight_nth_step_ahead_of_each_forecast(36)\n",
    "fig_param = m.plot_parameters()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extreme sparsity\n",
    "The lower we set `ar_sparsity`, the fewer non-zero weiths are fitted by the model. Here we set it to 1%, which should lead to a single non-zero lag.\n",
    "\n",
    "Note: Extreme values can lead to training instability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                                                                                  \r"
     ]
    }
   ],
   "source": [
    "m = NeuralProphet(\n",
    "    n_lags=6*12,\n",
    "    n_forecasts=3*12,\n",
    "    n_changepoints=0,\n",
    "    daily_seasonality=False,\n",
    "    weekly_seasonality=False,\n",
    "    learning_rate=0.01,\n",
    "    ar_sparsity=0.1,\n",
    ")\n",
    "metrics = m.fit(df, freq='5min')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 28088 characters]",
      "text/plain": [
       "<Figure size 720x432 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig_param = m.plot_parameters()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 28392 characters]",
      "text/plain": [
       "<Figure size 720x432 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "m = m.highlight_nth_step_ahead_of_each_forecast(1)\n",
    "fig_param = m.plot_parameters()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 28592 characters]",
      "text/plain": [
       "<Figure size 720x432 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "m = m.highlight_nth_step_ahead_of_each_forecast(36)\n",
    "fig_param = m.plot_parameters()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "np-dev",
   "language": "python",
   "name": "np-dev"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
