{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "93723569-e97b-4e07-9b7b-344f06f9dfa7",
   "metadata": {},
   "source": [
    "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ourownstory/neural_prophet/blob/master/tutorials/feature-use/crossvalidation.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f2af163-c378-4e01-8d5b-3def6194c29e",
   "metadata": {},
   "source": [
    "# Test and CrossValidate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "80400b6d-ca57-47ba-9dc5-0da3885ab6b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "if 'google.colab' in str(get_ipython()):\n",
    "    !pip install git+https://github.com/ourownstory/neural_prophet.git # may take a while\n",
    "    #!pip install neuralprophet # much faster, but may not have the latest upgrades/bugfixes\n",
    "    \n",
    "import pandas as pd\n",
    "from neuralprophet import NeuralProphet, set_log_level\n",
    "set_log_level(\"ERROR\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5e79793-8ebb-4d06-b021-82b49d107653",
   "metadata": {},
   "source": [
    "## Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "90e2ee7e-6798-4a0c-a37f-c3b7325d9c70",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_location = \"https://raw.githubusercontent.com/ourownstory/neuralprophet-data/main/datasets/\"\n",
    "# df = pd.read_csv(data_location + \"air_passengers.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24aaf32f-f3c4-4ac0-9317-122f4df932e7",
   "metadata": {},
   "source": [
    "## 1. Basic: Train and Test a model\n",
    "First, we show how to fit a model and evaluate it on a holdout set.\n",
    "\n",
    "Note: before making any actual forecasts, re-fit the model on all data available, else you are greatly reducing your forecast accuracy!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4e4871e-c8a5-48cb-b86c-de75adbbaf24",
   "metadata": {},
   "source": [
    "### 1.1 Train-Test evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "170af980-1830-4909-800a-b3e4f0ccc4f9",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                                                                                                                              \r"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SmoothL1Loss</th>\n",
       "      <th>MAE</th>\n",
       "      <th>MSE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.007121</td>\n",
       "      <td>31.413279</td>\n",
       "      <td>1440.266357</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   SmoothL1Loss        MAE          MSE\n",
       "0      0.007121  31.413279  1440.266357"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = NeuralProphet(seasonality_mode= \"multiplicative\", learning_rate = 0.1)\n",
    "\n",
    "df = pd.read_csv(data_location + \"air_passengers.csv\")\n",
    "df_train, df_test = m.split_df(df=df, freq=\"MS\", valid_p=0.2)\n",
    "\n",
    "metrics_train = m.fit(df=df_train, freq=\"MS\")\n",
    "metrics_test = m.test(df=df_test)\n",
    "\n",
    "metrics_test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bafa6a99-0df5-43cf-a235-0af2a51a4bcf",
   "metadata": {},
   "source": [
    "### 1.2 Bonus: Predict into future\n",
    "Before making any actual forecasts, re-fit the model on all data available, else you are greatly reducing your forecast accuracy!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "47d23260-ad96-4196-aba1-7eb9fd2e6b21",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                                                                                                                              \r"
     ]
    },
    {
     "data": {
      "image/png": "[encoded data, 49024 characters]\n",
      "text/plain": [
       "<Figure size 720x432 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "metrics_train2 = m.fit(df=df, freq=\"MS\")\n",
    "future = m.make_future_dataframe(df, periods=24, n_historic_predictions=48)\n",
    "forecast = m.predict(future)\n",
    "fig = m.plot(forecast)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7960ba2-6d3a-464c-837a-e8f844839d59",
   "metadata": {},
   "source": [
    "### 1.3 Bonus: Visualize training\n",
    "If you installed the `[live]` version of NeuralProphet, you can additionally visualize your training progress and spot any overfitting by evaluating every epoch.\n",
    "\n",
    "Note: Again, before making any predictions, re-fit the model with the entire data first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "db5b66bb-9734-49f9-af09-bc497509e040",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 57284 characters]\n",
      "text/plain": [
       "<Figure size 864x576 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                                                                                                                              \r"
     ]
    }
   ],
   "source": [
    "m = NeuralProphet(seasonality_mode=\"multiplicative\", learning_rate=0.1)\n",
    "\n",
    "df = pd.read_csv(data_location + \"air_passengers.csv\")\n",
    "df_train, df_test = m.split_df(df=df, freq=\"MS\", valid_p=0.2)\n",
    "\n",
    "metrics = m.fit(df=df_train, freq=\"MS\", validation_df=df_test, plot_live_loss=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "c84678f9-219f-4c7f-acca-3ff5b958855d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SmoothL1Loss</th>\n",
       "      <th>MAE</th>\n",
       "      <th>MSE</th>\n",
       "      <th>RegLoss</th>\n",
       "      <th>SmoothL1Loss_val</th>\n",
       "      <th>MAE_val</th>\n",
       "      <th>MSE_val</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>307</th>\n",
       "      <td>0.000317</td>\n",
       "      <td>6.178366</td>\n",
       "      <td>64.096661</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.005971</td>\n",
       "      <td>28.745131</td>\n",
       "      <td>1207.533936</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     SmoothL1Loss       MAE        MSE  RegLoss  SmoothL1Loss_val    MAE_val  \\\n",
       "307      0.000317  6.178366  64.096661      0.0          0.005971  28.745131   \n",
       "\n",
       "         MSE_val  \n",
       "307  1207.533936  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics.tail(1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3b0507f5-e4c6-4d97-845f-98a32cd0c64a",
   "metadata": {},
   "source": [
    "## 2. Time-series Cross-Validation\n",
    "Time-series crossvalidation is also known as rolling origin backtest. In the first fold, we start with some data to train the model, and evaluate over the next `fold_pct` data points. In the next fold, the previous evaluation data is added to training, and evaluation starts later (forecast orgin 'rolls' forward), again measuring accuracy over the next section of data. We repeat this, until the final folds evaluation data reaches the end of the data.\n",
    "\n",
    "Note: before making any actual forecasts, re-fit the model on all data available, else you are greatly reducing your forecast accuracy!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "fbf84c93-dc7c-406d-941a-87ae5b8adafa",
   "metadata": {},
   "outputs": [],
   "source": [
    "METRICS = ['SmoothL1Loss', 'MAE', 'MSE']\n",
    "params = {\"seasonality_mode\": \"multiplicative\", \"learning_rate\": 0.1}\n",
    "\n",
    "df = pd.read_csv(data_location + \"air_passengers.csv\")\n",
    "folds = NeuralProphet(**params).crossvalidation_split_df(df, freq=\"MS\", k=5, fold_pct=0.20, fold_overlap_pct=0.5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "b585c737-08f6-468e-890a-558348f6b9ee",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "                                                                                                                                                                                              \r"
     ]
    }
   ],
   "source": [
    "metrics_train = pd.DataFrame(columns=METRICS)\n",
    "metrics_test = pd.DataFrame(columns=METRICS)\n",
    "\n",
    "for df_train, df_test in folds:\n",
    "    m = NeuralProphet(**params)\n",
    "    train = m.fit(df=df_train, freq=\"MS\")\n",
    "    test = m.test(df=df_test)\n",
    "    metrics_train = metrics_train.append(train[METRICS].iloc[-1])\n",
    "    metrics_test = metrics_test.append(test[METRICS].iloc[-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "e09fc84b-3e11-47be-aa88-024b0c27c8a1",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SmoothL1Loss</th>\n",
       "      <th>MAE</th>\n",
       "      <th>MSE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.012356</td>\n",
       "      <td>22.827064</td>\n",
       "      <td>883.100308</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.012997</td>\n",
       "      <td>8.211850</td>\n",
       "      <td>508.476321</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.003793</td>\n",
       "      <td>10.655480</td>\n",
       "      <td>198.452499</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>0.034445</td>\n",
       "      <td>29.894016</td>\n",
       "      <td>1331.016846</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      SmoothL1Loss        MAE          MSE\n",
       "mean      0.012356  22.827064   883.100308\n",
       "std       0.012997   8.211850   508.476321\n",
       "min       0.003793  10.655480   198.452499\n",
       "max       0.034445  29.894016  1331.016846"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics_test.describe().loc[[\"mean\", \"std\", \"min\", \"max\"]]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d151730f-3354-47da-9d92-15abc3c87e2a",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 2 Using Benchmark Framework\n",
    "The benchmark framework is designed for a 2 phase evaluation pipeline. This is in most cases sufficient, particularly when using crossvalidation. For the remainder of this tutorial we will be using normal Train-Test or Cross-Validation evaluation setups.\n",
    "\n",
    "Note: The Benchmarking Framework does currently not properly support auto-regression or lagged covariates with multiple step ahead forecasts."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f23fd06b-2f0c-4f2c-92fa-5ad28e286d4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "from neuralprophet.benchmark import Dataset, NeuralProphetModel, SimpleExperiment, CrossValidationExperiment"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "87473a1c-e394-4b21-bba8-ae107671f114",
   "metadata": {},
   "source": [
    "### 2.1 SimpleExperiment\n",
    "Let's set up a train test experiment:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c7d050b7-2b8e-40d1-ba4d-e43e22a64c48",
   "metadata": {},
   "outputs": [],
   "source": [
    "ts = Dataset(df = pd.read_csv(data_location + \"air_passengers.csv\"), name = \"air_passengers\", freq = \"MS\")\n",
    "params = {\"seasonality_mode\": \"multiplicative\"}\n",
    "exp = SimpleExperiment(\n",
    "    model_class=NeuralProphetModel,\n",
    "    params=params,\n",
    "    data=ts,\n",
    "    metrics=[\"MASE\", \"RMSE\"],\n",
    "    test_percentage=25,\n",
    ")\n",
    "result_train, result_test = exp.run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "9b05621e-844f-4291-a81b-109e424829e7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'data': 'air_passengers',\n",
       " 'model': 'NeuralProphet',\n",
       " 'params': \"{'seasonality_mode': 'multiplicative'}\",\n",
       " 'MASE': 0.5662454805745694,\n",
       " 'RMSE': 28.829688840169517}"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# result_train\n",
    "result_test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c3e88e1-d1b9-46cc-bd8d-88dbf2f8fd93",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 2.2 CrossValidationExperiment\n",
    "Let's crossvalidate:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a5812bd-45d5-4a45-af9f-8dac25524e15",
   "metadata": {},
   "outputs": [],
   "source": [
    "ts = Dataset(df = pd.read_csv(data_location + \"air_passengers.csv\"), name = \"air_passengers\", freq = \"MS\")\n",
    "params = {\"seasonality_mode\": \"multiplicative\"}\n",
    "exp_cv = CrossValidationExperiment(\n",
    "    model_class=NeuralProphetModel,\n",
    "    params=params,\n",
    "    data=ts,\n",
    "    metrics=[\"MASE\", \"RMSE\"],\n",
    "    test_percentage=10,\n",
    "    num_folds=3,\n",
    "    fold_overlap_pct=0,\n",
    "  )\n",
    "result_train, result_test = exp_cv.run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "4a5d79f1-f61f-4c7d-9709-15535f1be8c9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'data': 'air_passengers',\n",
       " 'model': 'NeuralProphet',\n",
       " 'params': \"{'seasonality_mode': 'multiplicative'}\",\n",
       " 'MASE': [0.5044070952378811, 0.3214836090858653, 0.6877225952888634],\n",
       " 'RMSE': [21.220949835639722, 16.892885303566555, 32.749710284868435]}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "result_test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bbd37a7c-17eb-4f6f-a7c8-f4ac15c41183",
   "metadata": {
    "tags": []
   },
   "source": [
    "## 3. Advanced: 3-Phase Train, Validate and Test procedure\n",
    "Finally, in 3.1 and 3.2, we will do a 3-part data split to do a proper training, validation and test evaluation of your model. This setup is used if you do not want to bias your performance evaluation by your manual hyperparameter tuning. this is, however not common when working with time series, unless you work in academia. Crossvalidation is usually more than adequate to evaluate your model performance.\n",
    "\n",
    "If you are confused by this, simply ignore this section and continue your forecasting life. Or if you got curious, read up on how to evaluate machine learning models to level up your skills. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "44add497-fd4b-4926-9338-9b507bbec4e3",
   "metadata": {},
   "source": [
    "### 3.1 Train, Validate and Test evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e342fef-b57d-4ab8-8550-f9311cb0e032",
   "metadata": {},
   "outputs": [],
   "source": [
    "m = NeuralProphet(seasonality_mode= \"multiplicative\", learning_rate = 0.1)\n",
    "\n",
    "df = pd.read_csv(data_location + \"air_passengers.csv\")\n",
    "# create a test holdout set:\n",
    "df_train_val, df_test = m.split_df(df=df, freq=\"MS\", valid_p=0.2)\n",
    "# create a validation holdout set:\n",
    "df_train, df_val = m.split_df(df=df_train_val, freq=\"MS\", valid_p=0.2)\n",
    "\n",
    "# fit a model on training data and evaluate on validation set.\n",
    "metrics_train1 = m.fit(df=df_train, freq=\"MS\")\n",
    "metrics_val = m.test(df=df_val)\n",
    "\n",
    "# refit model on training and validation data and evaluate on test set.\n",
    "metrics_train2 = m.fit(df=df_train_val, freq=\"MS\")\n",
    "metrics_test = m.test(df=df_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "635e284c-b3b7-477c-ada0-9a131912137f",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SmoothL1Loss</th>\n",
       "      <th>MAE</th>\n",
       "      <th>MSE</th>\n",
       "      <th>split</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>324</th>\n",
       "      <td>0.000368</td>\n",
       "      <td>5.348336</td>\n",
       "      <td>46.359198</td>\n",
       "      <td>train1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>649</th>\n",
       "      <td>0.000571</td>\n",
       "      <td>6.835680</td>\n",
       "      <td>72.002425</td>\n",
       "      <td>train2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.005073</td>\n",
       "      <td>18.293804</td>\n",
       "      <td>639.260681</td>\n",
       "      <td>validate</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.002529</td>\n",
       "      <td>15.051163</td>\n",
       "      <td>318.605957</td>\n",
       "      <td>test</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     SmoothL1Loss        MAE         MSE     split\n",
       "324      0.000368   5.348336   46.359198    train1\n",
       "649      0.000571   6.835680   72.002425    train2\n",
       "0        0.005073  18.293804  639.260681  validate\n",
       "0        0.002529  15.051163  318.605957      test"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics_train1[\"split\"]  = \"train1\"\n",
    "metrics_train2[\"split\"]  = \"train2\"\n",
    "metrics_val[\"split\"] = \"validate\"\n",
    "metrics_test[\"split\"] = \"test\"\n",
    "metrics_train1.tail(1).append([metrics_train2.tail(1), metrics_val, metrics_test]).drop(columns=['RegLoss'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "56864317-2b2f-4332-8d76-40a40e287b9d",
   "metadata": {
    "tags": []
   },
   "source": [
    "### 3.2 Train, Cross-Validate and Cross-Test evaluation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "dac90c4f-63e7-42f2-b49e-f6b3177e070f",
   "metadata": {},
   "outputs": [],
   "source": [
    "METRICS = ['SmoothL1Loss', 'MAE', 'MSE']\n",
    "params = {\"seasonality_mode\": \"multiplicative\", \"learning_rate\": 0.1}\n",
    "\n",
    "df = pd.read_csv(data_location + \"air_passengers.csv\")\n",
    "folds_val, folds_test = NeuralProphet(**params).double_crossvalidation_split_df(df, freq=\"MS\", k=5, valid_pct=0.10, test_pct=0.10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "96d8154e-c77a-4836-ad92-9b205e88b268",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "metrics_train1 = pd.DataFrame(columns=METRICS)\n",
    "metrics_val = pd.DataFrame(columns=METRICS)\n",
    "for df_train1, df_val in folds_val:\n",
    "    m = NeuralProphet(**params)\n",
    "    train1 = m.fit(df=df_train, freq=\"MS\")\n",
    "    val = m.test(df=df_val)\n",
    "    metrics_train1 = metrics_train1.append(train1[METRICS].iloc[-1])\n",
    "    metrics_val = metrics_val.append(val[METRICS].iloc[-1])\n",
    "\n",
    "metrics_train2 = pd.DataFrame(columns=METRICS)\n",
    "metrics_test = pd.DataFrame(columns=METRICS)\n",
    "for df_train2, df_test in folds_test:\n",
    "    m = NeuralProphet(**params)\n",
    "    train2 = m.fit(df=df_train2, freq=\"MS\")\n",
    "    test = m.test(df=df_test)\n",
    "    metrics_train2 = metrics_train2.append(train2[METRICS].iloc[-1])\n",
    "    metrics_test = metrics_test.append(test[METRICS].iloc[-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "c5ea1cbc-fbb2-46b1-9a43-4547f8b6113a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SmoothL1Loss</th>\n",
       "      <th>MAE</th>\n",
       "      <th>MSE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.000269</td>\n",
       "      <td>6.724876</td>\n",
       "      <td>74.517579</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.000023</td>\n",
       "      <td>0.090767</td>\n",
       "      <td>1.726542</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      SmoothL1Loss       MAE        MSE\n",
       "mean      0.000269  6.724876  74.517579\n",
       "std       0.000023  0.090767   1.726542"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics_train2.describe().loc[[\"mean\", \"std\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "13f96119-df3c-4a0d-adeb-f2816ff88587",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SmoothL1Loss</th>\n",
       "      <th>MAE</th>\n",
       "      <th>MSE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.009112</td>\n",
       "      <td>29.609479</td>\n",
       "      <td>1148.171173</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.007296</td>\n",
       "      <td>13.811697</td>\n",
       "      <td>919.264304</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      SmoothL1Loss        MAE          MSE\n",
       "mean      0.009112  29.609479  1148.171173\n",
       "std       0.007296  13.811697   919.264304"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics_val.describe().loc[[\"mean\", \"std\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "6c6327f8-bcb3-4454-bc9f-a1fe26f8bea6",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>SmoothL1Loss</th>\n",
       "      <th>MAE</th>\n",
       "      <th>MSE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>0.001103</td>\n",
       "      <td>14.406253</td>\n",
       "      <td>295.007996</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>0.001310</td>\n",
       "      <td>7.844408</td>\n",
       "      <td>337.153951</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      SmoothL1Loss        MAE         MSE\n",
       "mean      0.001103  14.406253  295.007996\n",
       "std       0.001310   7.844408  337.153951"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "metrics_test.describe().loc[[\"mean\", \"std\"]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be076455-5393-4628-9079-7efa60b8e404",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "b0fa6594d8f4cbf19f97940f81e996739fb7646882a419484c72d19e05852a7e"
  },
  "kernelspec": {
   "display_name": "np-dev",
   "language": "python",
   "name": "np-dev"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
