{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# **Getting Started with NETS**\n",
    "\n",
    "**NETS** is a vanilla Deep Learning framework, made using only **NumPy**.\n",
    "This project was first introduced as an assignment I made at the [University of Oslo](https://www.uio.no/studier/emner/matnat/ifi/IN5400/) and [Stanford University](http://cs231n.stanford.edu/syllabus.html)\n",
    ".\n",
    "However, this project was recreated to make it *object-oriented like*.\n",
    "**NETS** was highly inspired from [PyTorch](https://pytorch.org/) and [TensorFlow](https://www.tensorflow.org/)\n",
    "\n",
    "Requirements :\n",
    "\n",
    "- Numpy\n",
    "- Json (Optional)\n",
    "- Time (Optional)\n",
    "- Pandas (Optional)\n",
    "- Scipy (Optional)\n",
    "- Scikit-Learn (Optional)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 0. Set Up\n",
    "\n",
    "Let's load the packages we need to run our tests:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import random as rd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# This is a bit of magic to make matplotlib figures appear inline in the notebook\n",
    "# rather than in a new window.\n",
    "%matplotlib inline\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots\n",
    "plt.rcParams['image.interpolation'] = 'nearest'\n",
    "plt.style.use('seaborn-darkgrid')\n",
    "# Change the font ?\n",
    "font = {'size'   : 15}\n",
    "plt.rc('font', **font)\n",
    "\n",
    "# Some more magic so that the notebook will reload external python modules;\n",
    "# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's fix the seed\n",
    "\n",
    "SEED = 42\n",
    "rd.seed(SEED)\n",
    "np.random.seed(SEED)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Tensor\n",
    "\n",
    "A tensor is multi-dimensional array, similar to NumPy arrays.\n",
    "Their particularity is their capability to stock previous gradients and operations.\n",
    "Their architecture was highly inspired from *PyTorch* documentation.\n",
    "\n",
    "In *PyTorch*, you create tensors using the tensor built-in method ``tensor`` or use ``Tensor`` class directly (it is not recommended to instantiate a tensor this way however).\n",
    "*Nets* does not have (yet) built-in functions, so you will need to use the ``Tensor`` class to create a tensor.\n",
    "\n",
    "\n",
    "## 1.1. Basic operations\n",
    "\n",
    "Tensors supports basic mathematical operations. All these operations are defined in the ``nets/ops.py`` module. Here is a list of operations currently supported:\n",
    "\n",
    "- ``add`` which add to tensors (broadcasting is supported),\n",
    "- ``subtract`` which subtract to tensors (broadcasting is supported),\n",
    "- ``multiply`` which add to tensors with the same shape,\n",
    "- ``dot`` which compute the dot product ``@`` of two matrices,\n",
    "- ``exp`` which compute the element-wise exponentiation of a tensor.\n",
    "- ``sum`` which sum up all elements in a tensor,\n",
    "- ``transpose`` which transpose a tensor. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "t1 =\n",
      "Tensor([[1, 3],\n",
      "        [5, 7]])\n",
      "t2 =\n",
      "Tensor([[2, 4],\n",
      "        [6, 8]])\n",
      "\n",
      "Some basic operations\n",
      "t1 + t2: \n",
      "Tensor([[ 3,  7],\n",
      "        [11, 15]])\n",
      "t1 - t2: \n",
      "Tensor([[-1, -1],\n",
      "        [-1, -1]])\n",
      "t1 * t2: \n",
      "Tensor([[ 2, 12],\n",
      "        [30, 56]])\n",
      "t1 @ t2: \n",
      "Tensor([[20, 28],\n",
      "        [52, 76]])\n",
      "t1 ** 2: \n",
      "Tensor([[ 1,  9],\n",
      "        [25, 49]])\n",
      "t1 / t2: \n",
      "Tensor([[0.5000, 0.7500],\n",
      "        [0.8333, 0.8750]])\n",
      "t1 / 10: \n",
      "Tensor([[0.1, 0.3],\n",
      "        [0.5, 0.7]])\n"
     ]
    }
   ],
   "source": [
    "import nets\n",
    "\n",
    "t1 = nets.Tensor([[1, 3], \n",
    "                  [5, 7]])\n",
    "t2 = nets.Tensor([[2, 4], \n",
    "                  [6, 8]])\n",
    "\n",
    "print(f\"t1 =\\n{t1}\")\n",
    "print(f\"t2 =\\n{t2}\")\n",
    "\n",
    "print(\"\\nSome basic operations\")\n",
    "print(f\"t1 + t2: \\n{t1 + t2}\")\n",
    "print(f\"t1 - t2: \\n{t1 - t2}\")\n",
    "print(f\"t1 * t2: \\n{t1 *t2}\")\n",
    "print(f\"t1 @ t2: \\n{t1 @ t2}\")\n",
    "print(f\"t1 ** 2: \\n{t1 ** 2}\")\n",
    "print(f\"t1 / t2: \\n{t1 / t2}\")\n",
    "print(f\"t1 / 10: \\n{t1 / 10}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2. Gradients\n",
    "\n",
    "*Nets* uses a custom *autograd* system, made with numpy.\n",
    "Some vanilla architectures do not depends on this functionality however, like ``DNN`` networks.\n",
    "All these information will be detailed in the model's section.\n",
    "\n",
    "As you may have seen, there is a ``requires_grad`` set to ``False`` by default when we create a tensor.\n",
    "This attribute attributes works similarly as **PyTorch**'s attribute. If set to ``True``,\n",
    "previous gradients will be registered and saved in this tensor, in the ``_hooks`` attribute.\n",
    "This attribute is basically a list containing all previous gradients.\n",
    "That is, when calling the ``backward`` method on this tensor with an upstream gradient,\n",
    "it will propagate through all previous gradients.\n",
    "\n",
    "Let's see some basic examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "t1 =\n",
      "Tensor([1, 3], requires_grad=True)\n",
      "t2 =\n",
      "Tensor([2, 4], requires_grad=True)\n",
      "\n",
      "Operation:\n",
      "t3 = t1 + t2 + 4\n",
      "t4 = t3 * t2\n",
      "\n",
      "t3 =\n",
      "Tensor([ 7, 11], requires_grad=True)\n",
      "t4 =\n",
      "Tensor([14, 44], requires_grad=True)\n",
      "\n",
      "Before backpropagation\n",
      "t1 gradient: Tensor([0., 0.])\n",
      "t2 gradient: Tensor([0., 0.])\n",
      "t3 gradient: Tensor([0., 0.])\n",
      "t4 gradient: Tensor([0., 0.])\n",
      "\n",
      "After backpropagation\n",
      "t1 gradient: Tensor([-2.,  8.])\n",
      "t2 gradient: Tensor([-9., 30.])\n",
      "t3 gradient: Tensor([-2.,  8.])\n",
      "t4 gradient: Tensor([-1.,  2.])\n"
     ]
    }
   ],
   "source": [
    "t1 = nets.Tensor([1, 3], requires_grad=True)\n",
    "t2 = nets.Tensor([2, 4], requires_grad=True)\n",
    "\n",
    "# Some operations\n",
    "t3 = t1 + t2 + 4\n",
    "t4 = t3 * t2\n",
    "\n",
    "print(f\"t1 =\\n{t1}\")\n",
    "print(f\"t2 =\\n{t2}\")\n",
    "\n",
    "print(\"\\nOperation:\")\n",
    "print(\"t3 = t1 + t2 + 4\")\n",
    "print(\"t4 = t3 * t2\")\n",
    "\n",
    "print(f\"\\nt3 =\\n{t3}\")\n",
    "print(f\"t4 =\\n{t4}\")\n",
    "\n",
    "print(\"\\nBefore backpropagation\")\n",
    "print(f\"t1 gradient: {t1.grad}\")\n",
    "print(f\"t2 gradient: {t2.grad}\")\n",
    "print(f\"t3 gradient: {t3.grad}\")\n",
    "print(f\"t4 gradient: {t4.grad}\")\n",
    "\n",
    "# Upstream gradient\n",
    "grad = nets.Tensor([-1, 2])\n",
    "\n",
    "# Back-propagation\n",
    "t4.backward(grad)\n",
    "\n",
    "print(\"\\nAfter backpropagation\")\n",
    "print(f\"t1 gradient: {t1.grad}\")\n",
    "print(f\"t2 gradient: {t2.grad}\")\n",
    "print(f\"t3 gradient: {t3.grad}\")\n",
    "print(f\"t4 gradient: {t4.grad}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Autograd\n",
    "\n",
    "The backward pass is called for computing the gradients and create a computational graph from all previous operations. \n",
    "This backward pass can be called using the ``backward`` method of a ``Module``. \n",
    "In this case, you will need to write the computational on your own, as the gradients' equations depends on a model. \n",
    "This *Vanilla* back-propagation is implemented for standard models, like ``DNN``, ``CNN`` and ``RNN``.\n",
    "However, you won't be able to mix them using this technique unless you change some parts of the ``backward`` method.\n",
    "\n",
    "Or, you can use the *autograd* system. As ``Tensor`` keeps track of the computational graph in their ``hooks``, \n",
    "it is easier to compute the back-propagation. The back-propagation is decomposed in elementary operations (+, -, /, \\*, exp) \n",
    "and the gradient given from a ``Loss`` function is then transferred through the computational graph."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example\n",
    "\n",
    "With the backward pass, you can minimize any basic functions (functions that can be decomposed ito elementary operations). As the ``autograd`` system, record the gradients, you can determine the slope at this point and adjust your inputs by a coefficient, called learning rate $l_r$.\n",
    "\n",
    "This is maybe a lot of information and a lot of text, so let's try to visualize how we can use gradients to minimize a function instead.\n",
    "\n",
    "The function we will try to minimize is the so called *three hump camel* function:\n",
    "\n",
    "$$\n",
    "\\forall x_1, x_2 \\in \\mathbb{R}, \\quad f(x_1, x_2) = 2 x_1^2 - 1.05 x_1^4 + \\frac{x_1^6}{6} + x_1 x_2 + x_2^2\n",
    "$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<mpl_toolkits.mplot3d.art3d.Poly3DCollection at 0x1f736986b48>"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 144588 characters]\n",
      "text/plain": [
       "<Figure size 720x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "\n",
    "fig = plt.figure()\n",
    "ax = fig.gca(projection='3d')\n",
    "\n",
    "# Make data\n",
    "camel = lambda x,y: (2*x**2 - 1.05*x**4 + ((x**6)/6) + x*y + y**2)\n",
    "X = np.linspace(-1, 1, 100)\n",
    "Y = np.linspace(-1, 1, 100)\n",
    "X, Y = np.meshgrid(X, Y)\n",
    "Z = camel(X, Y)\n",
    "\n",
    "# Plot the surface\n",
    "ax.view_init(30, 105)\n",
    "ax.plot_surface(X, Y, Z, cmap='coolwarm', linewidth=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1. PyTorch implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch:   49 | loss: 3.63E-08"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "from torch.autograd import Variable\n",
    "\n",
    "# Random x points\n",
    "data = rd.uniform(-1, 1)\n",
    "x = torch.tensor(data, requires_grad=True)\n",
    "# Random y points\n",
    "data = rd.uniform(-1, 1)\n",
    "y = torch.tensor(data, requires_grad=True)\n",
    "\n",
    "# Wrap X, Y in Variable for autograd\n",
    "x = Variable(x, requires_grad=True)\n",
    "y = Variable(y, requires_grad=True)\n",
    "\n",
    "# Learning rate\n",
    "lr = 0.1\n",
    "\n",
    "# Define the update rule\n",
    "optimizer = torch.optim.SGD([x, y], lr=.1, momentum=0)\n",
    "\n",
    "# Keep track of the loss\n",
    "torch_history = []\n",
    "torch_points_history = []\n",
    "\n",
    "for i in range(50):\n",
    "    # The gradients accumulates, we need to clear that at each epochs\n",
    "    optimizer.zero_grad()\n",
    "    \n",
    "    # outputs, can be seen as \"predictions\"\n",
    "    z = camel(x, y)\n",
    "    \n",
    "    # Compute the loss (sum of all values)\n",
    "    # As the minimum is at (0, 0), the lower the loss is, the closest we are to this minimum\n",
    "    loss = z.sum()\n",
    "    \n",
    "    # Get the gradients\n",
    "    loss.backward()\n",
    "    \n",
    "    # Update the points\n",
    "    optimizer.step()\n",
    "\n",
    "    # Add the loss to the history and display the current loss in the console\n",
    "    torch_history.append(loss.item())\n",
    "    torch_points_history.append((x.item(), y.item(), z.item()))\n",
    "    print(f\"\\repoch: {i:4d} | loss: {loss.item():1.2E}\", end=\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.2. NETS implementation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch:   49 | loss: 3.61E-09"
     ]
    }
   ],
   "source": [
    "from nets.optim import SGD\n",
    "\n",
    "# Random x points\n",
    "data = rd.uniform(-1, 1)\n",
    "xn = nets.Parameter(data)\n",
    "# Random y points\n",
    "data = rd.uniform(-1, 1)\n",
    "yn = nets.Parameter(data)\n",
    "\n",
    "# Now we have generated points (X, Y) randomly\n",
    "\n",
    "# Learning rate\n",
    "lr = 0.1\n",
    "\n",
    "optimizer = SGD([xn, yn], lr=lr, momentum=0)\n",
    "\n",
    "# Keep track of the loss\n",
    "nets_history = []\n",
    "nets_points_history = []\n",
    "\n",
    "# Run the simulation 50 times\n",
    "for i in range(50):\n",
    "    # The gradients accumulates, we need to clear that at each epochs\n",
    "    optimizer.zero_grad()\n",
    "    \n",
    "    # outputs, can be seen as \"predictions\"\n",
    "    zn = camel(xn, yn)\n",
    "    \n",
    "    # Compute the loss (sum of all values)\n",
    "    # As the minimum is at (0, 0), the lower the loss is, the closest we are to this minimum\n",
    "    loss = zn.sum()  # is a 0-tensor\n",
    "    \n",
    "    # Get the gradients\n",
    "    loss.backward()\n",
    "    \n",
    "    # Update the points\n",
    "    optimizer.step()\n",
    "    \n",
    "    # Add the loss to the history and display the current loss in the console\n",
    "    nets_history.append(loss.item())\n",
    "    nets_points_history.append((xn.item(), yn.item(), zn.item()))\n",
    "    print(f\"\\repoch: {i:4d} | loss: {loss.item():1.2E}\", end=\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x1f7391df908>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 35292 characters]\n",
      "text/plain": [
       "<Figure size 720x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(nets_history, label='NETS')\n",
    "plt.plot(torch_history, label='PyTorch')\n",
    "plt.title(\"Minimizing the three hump camel function\")\n",
    "plt.ylabel('loss')\n",
    "plt.xlabel('epochs')\n",
    "plt.legend(loc=0, frameon=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x1f73922d9c8>"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 180916 characters]\n",
      "text/plain": [
       "<Figure size 720x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure()\n",
    "ax = fig.gca(projection='3d')\n",
    "\n",
    "# Plot the surface\n",
    "ax.view_init(60, 105)\n",
    "ax.plot_surface(X, Y, Z, cmap='coolwarm', linewidth=0)\n",
    "\n",
    "# Plot the history of the minimization process\n",
    "x_torch, y_torch, z_torch = list(zip(*torch_points_history))\n",
    "x_nets, y_nets, z_nets = list(zip(*nets_points_history))\n",
    "\n",
    "ax.plot(x_torch, y_torch, z_torch, label=\"PyTorch\")\n",
    "ax.plot(x_nets, y_nets, z_nets, label=\"NETS\")\n",
    "#Plot a legend\n",
    "plt.legend(loc=1, frameon=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x1f7392d3108>"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 25112 characters]\n",
      "text/plain": [
       "<Figure size 720x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.scatter(x_nets, y_nets, label='NETS')\n",
    "plt.scatter(x_torch, y_torch, label='PyTorch')\n",
    "plt.scatter(0, 0, color='r', label='minimum')\n",
    "plt.title(\"Minimums\")\n",
    "plt.xlabel(\"x\")\n",
    "plt.ylabel(\"y\")\n",
    "plt.xlim(-0.05, 0.05)\n",
    "plt.xlim(-0.05, 0.05)\n",
    "plt.legend(frameon=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The minimums found are all located close to $(0, 0)$, which is the global minimum of $f$. It worked !"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Build a model\n",
    "\n",
    "A model is a ``Module``subclass, where biases, weights and parameters transformations are computed. All modules have a ``forward`` method, that MUST be overwritten. This method will compute the forward propagation from an input tensor, and compute the transformation. If using the ``autograd`` system, no back-propagation need to be added. However, if you prefer to manually compute the gradients, you will need to override the ``backward`` method.\n",
    "\n",
    "## 3.1. Sequential\n",
    "\n",
    "There are two approaches when it comes to building a model. The first one, maybe the less convenient, is to use the ``Sequential`` class or ``add`` method. When building a sequential model, the ``forward`` pass is just a succession of inner forward loops. For example, if we have objects that have 10 *attributes* and we want to classify them in 5 classes, we can try the following model:\n",
    "\n",
    "```\n",
    "Linear(10, 100) -> Tanh() -> Linear(100, 100) -> Tanh() -> Linear(100, 5)\n",
    "```\n",
    "\n",
    "the forward pass will call the ``forward`` method of:\n",
    "\n",
    "- Linear(10, 100),\n",
    "- Tanh(),\n",
    "- Linear(100, 100),\n",
    "- Tanh(),\n",
    "- Linear(100, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\home\\Arthur\\Documents\\Informatique\\GitHub\\nets\\nets\\nn\\module\\linear.py:33: DeprecationWarning: Call to deprecated function init_params (weights and biases will be initialized in another function soon. This method will be then deprecated in the next update).\n",
      "  self.init_params()\n",
      "D:\\home\\Arthur\\Documents\\Informatique\\GitHub\\nets\\nets\\nn\\module\\linear.py:33: DeprecationWarning: Call to deprecated function init_params (weights and biases will be initialized in another function soon. This method will be then deprecated in the next update).\n",
      "  self.init_params()\n",
      "D:\\home\\Arthur\\Documents\\Informatique\\GitHub\\nets\\nets\\nn\\module\\linear.py:33: DeprecationWarning: Call to deprecated function init_params (weights and biases will be initialized in another function soon. This method will be then deprecated in the next update).\n",
      "  self.init_params()\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Sequential(\n",
       "   (0): Linear(input_dim=10, output_dim=100, bias=True)\n",
       "   (1): Tanh()\n",
       "   (2): Linear(input_dim=100, output_dim=100, bias=True)\n",
       "   (3): Tanh()\n",
       "   (4): Linear(input_dim=100, output_dim=5, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nets.nn as nn\n",
    "\n",
    "#Build a sequential model\n",
    "model = nn.Sequential(nn.Linear(10, 100),\n",
    "                      nn.Tanh(),\n",
    "                      nn.Linear(100, 100),\n",
    "                      nn.Tanh(),\n",
    "                      nn.Linear(100, 5))\n",
    "\n",
    "# Let's check the architecture\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can of course add manually a module to an existing model using ``add``:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\home\\Arthur\\Documents\\Informatique\\GitHub\\nets\\nets\\nn\\module\\linear.py:33: DeprecationWarning: Call to deprecated function init_params (weights and biases will be initialized in another function soon. This method will be then deprecated in the next update).\n",
      "  self.init_params()\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Sequential(\n",
       "   (0): Linear(input_dim=10, output_dim=100, bias=True)\n",
       "   (1): Tanh()\n",
       "   (2): Linear(input_dim=100, output_dim=100, bias=True)\n",
       "   (3): Tanh()\n",
       "   (4): Linear(input_dim=100, output_dim=5, bias=True)\n",
       "   (5): Tanh()\n",
       "   (6): Linear(input_dim=5, output_dim=5, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.add(nn.Tanh(), nn.Linear(5, 5))\n",
    "\n",
    "# Let's check the architecture\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2. Class Instance\n",
    "\n",
    "The second initialization is the most popular (used by **PyTorch** and **TensorFlow**). It consists by creating a ``Model`` class that inherits from ``Module`` and override the ``forward`` pass."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\home\\Arthur\\Documents\\Informatique\\GitHub\\nets\\nets\\nn\\module\\linear.py:33: DeprecationWarning: Call to deprecated function init_params (weights and biases will be initialized in another function soon. This method will be then deprecated in the next update).\n",
      "  self.init_params()\n",
      "D:\\home\\Arthur\\Documents\\Informatique\\GitHub\\nets\\nets\\nn\\module\\linear.py:33: DeprecationWarning: Call to deprecated function init_params (weights and biases will be initialized in another function soon. This method will be then deprecated in the next update).\n",
      "  self.init_params()\n",
      "D:\\home\\Arthur\\Documents\\Informatique\\GitHub\\nets\\nets\\nn\\module\\linear.py:33: DeprecationWarning: Call to deprecated function init_params (weights and biases will be initialized in another function soon. This method will be then deprecated in the next update).\n",
      "  self.init_params()\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Model(\n",
       "   (layer1): Linear(input_dim=10, output_dim=100, bias=True)\n",
       "   (layer2): Linear(input_dim=100, output_dim=100, bias=True)\n",
       "   (layer3): Linear(input_dim=100, output_dim=5, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import nets\n",
    "import nets.nn as nn\n",
    "\n",
    "class Model(nn.Module):\n",
    "    \"\"\"\n",
    "    Create your own model.\n",
    "    The attributes should be the submodel used during the forward pass.\n",
    "    You don't have to necessary affect the activation function, unless you want to set a manual backward pass.\n",
    "    \"\"\"\n",
    "    \n",
    "    def __init__(self, input_dim, hidden_dim, output_dim):\n",
    "        # Initialization\n",
    "        \n",
    "        super().__init__()               # Don't forget to add this line\n",
    "        self.layer1 = nn.Linear(input_dim, hidden_dim)\n",
    "        self.layer2 = nn.Linear(hidden_dim, hidden_dim)\n",
    "        self.layer3 = nn.Linear(hidden_dim, output_dim)\n",
    "        \n",
    "    def forward(self, inputs):\n",
    "        # Forward pass\n",
    "        \n",
    "        out1 = nets.tanh(self.layer1(inputs))\n",
    "        out2 = nets.tanh(self.layer2(out1))\n",
    "        return self.layer3(out2)\n",
    "    \n",
    "model = Model(10, 100, 5)\n",
    "\n",
    "# Let's check the architecture\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Forward pass\n",
    "\n",
    "As seen before, the ``forward`` pass is essential to compute the predicted classes to solve a problem. For an input tensor, the first index should corresponds to the ``BATCH_SIZE`` $N$ and the second to the ``input_dim`` dimension. The result is a tensor of shape $(N, C)$ where $C$ stands for the number of classes (``output_dim`` in the previous model)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data shape: (50, 10)\n"
     ]
    }
   ],
   "source": [
    "# Build inputs tensor\n",
    "# The data is between [-10, 10]\n",
    "\n",
    "BATCH_SIZE = 50\n",
    "data = (np.random.rand(BATCH_SIZE, 10) * 20 - 10)\n",
    "inputs = nets.Tensor(data)\n",
    "\n",
    "print(f'data shape: {inputs.shape}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "predictions' shape: (50, 5)\n"
     ]
    }
   ],
   "source": [
    "predictions = model.forward(inputs)\n",
    "\n",
    "print(f\"predictions' shape: {predictions.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. All together\n",
    "\n",
    "## Example: XOR\n",
    "\n",
    "Let's try to solve the XOR problem. This problem consist to classify two classes distributed as follow:\n",
    "\n",
    "![XOR](img/xor.png)\n",
    "\n",
    "This distribution **is not linearly separable**, so we will need non-linearity transformations in our model to be able to correctly classify each elements.\n",
    "\n",
    "## 5.1. Generating the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data shape: (200, 2)\n",
      "labels shape: (200, 2)\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import nets\n",
    "\n",
    "# [1, 1] quarter\n",
    "d1 = np.random.rand(50, 2)\n",
    "# [-1, -1] quarter\n",
    "d2 = (np.random.rand(50, 2) - 1)\n",
    "# Labels\n",
    "l1 = np.ones(len(d1) + len(d2))\n",
    "l1 = np.column_stack((l1, np.zeros(len(d1) + len(d2))))\n",
    "\n",
    "# [-1, 1] quarter\n",
    "d3 = np.random.rand(50, 2)\n",
    "d3[:,0] -= np.ones(50)\n",
    "# [1, -1] quarter\n",
    "d4 = np.random.rand(50, 2)\n",
    "d4[:,1] -= np.ones(50)\n",
    "# Labels\n",
    "l2 = np.zeros(len(d3) + len(d4))\n",
    "l2 = np.column_stack((l2, np.ones(len(d1) + len(d2))))\n",
    "\n",
    "\n",
    "# All the data\n",
    "data = nets.Tensor(np.concatenate((d1, d2, d3, d4)))\n",
    "labels = np.concatenate((l1, l2)).astype(int)\n",
    "# labels = np.argmax(labels, axis=1)\n",
    "labels = nets.Tensor(labels)\n",
    "# And all labels :\n",
    "# [1, 0] one hot encoded: 0 -> blue\n",
    "# [0, 1] one hot encoded: 1 -> red\n",
    "\n",
    "print(f\"data shape: {data.shape}\")\n",
    "print(f\"labels shape: {labels.shape}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.legend.Legend at 0x1f7392ebe88>"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 71052 characters]\n",
      "text/plain": [
       "<Figure size 720x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "scatter = plt.scatter(data.data[:,0], data.data[:,1], s=40, c=np.argmax(labels.data, axis=1), cmap='coolwarm')\n",
    "plt.legend(*scatter.legend_elements(), loc=1, frameon=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.2. Building a model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "D:\\home\\Arthur\\Documents\\Informatique\\GitHub\\nets\\nets\\nn\\module\\linear.py:33: DeprecationWarning: Call to deprecated function init_params (weights and biases will be initialized in another function soon. This method will be then deprecated in the next update).\n",
      "  self.init_params()\n",
      "D:\\home\\Arthur\\Documents\\Informatique\\GitHub\\nets\\nets\\nn\\module\\linear.py:33: DeprecationWarning: Call to deprecated function init_params (weights and biases will be initialized in another function soon. This method will be then deprecated in the next update).\n",
      "  self.init_params()\n",
      "D:\\home\\Arthur\\Documents\\Informatique\\GitHub\\nets\\nets\\nn\\module\\linear.py:33: DeprecationWarning: Call to deprecated function init_params (weights and biases will be initialized in another function soon. This method will be then deprecated in the next update).\n",
      "  self.init_params()\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "XOR(\n",
       "   (layer1): Linear(input_dim=2, output_dim=10, bias=True)\n",
       "   (layer2): Linear(input_dim=10, output_dim=10, bias=True)\n",
       "   (layer3): Linear(input_dim=10, output_dim=2, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "class XOR(nn.Module):\n",
    "\n",
    "    def __init__(self, input_dim, hidden_dim, output_dim):\n",
    "        super().__init__()\n",
    "        self.layer1 = nn.Linear(input_dim, hidden_dim)\n",
    "        self.layer2 = nn.Linear(hidden_dim, hidden_dim)\n",
    "        self.layer3 = nn.Linear(hidden_dim, output_dim)\n",
    "\n",
    "    def forward(self, inputs):\n",
    "        out1 = nets.tanh(self.layer1(inputs))\n",
    "        out2 = nets.tanh(self.layer2(out1))\n",
    "        out3 = self.layer3(out2)\n",
    "        return out3\n",
    "    \n",
    "model = XOR(2, 10, 2)\n",
    "\n",
    "# Let's check the architecture\n",
    "model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.3. Criterion & Optimizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nets.optim import SGD, Adam, RMSprop\n",
    "from nets.nn import CrossEntropyLoss, MSELoss\n",
    "from nets.nn.utils import one_hot\n",
    "\n",
    "# How much do we update the parameters\n",
    "LEARNING_RATE = 0.01\n",
    "\n",
    "optimizer = Adam(model.parameters(), lr=LEARNING_RATE)\n",
    "criterion = CrossEntropyLoss()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.4. Training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 4999 | loss: 6.602E-06"
     ]
    }
   ],
   "source": [
    "EPOCHS = 5000\n",
    "\n",
    "history = []\n",
    "for epoch in range(EPOCHS):\n",
    "    # Gradients accumulates, therefore we need to set them to zero at each iteration\n",
    "    model.zero_grad()\n",
    "    # Predictions\n",
    "    predictions = model(data)\n",
    "    loss = criterion(predictions, labels)\n",
    "    # Compute the gradient\n",
    "    loss.backward()\n",
    "    # Update the parameters\n",
    "    optimizer.step()\n",
    "    # Record the loss for plotting\n",
    "    history.append(loss.item())\n",
    "    print(f\"\\repoch: {epoch} | loss: {loss.item():1.3E}\", end=\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x1f74618d848>]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "[encoded data, 18460 characters]\n",
      "text/plain": [
       "<Figure size 720x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.5. Testing\n",
    "\n",
    "Again, we need to generate randomly unseen data (testing data)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "data shape: (200, 2)\n",
      "labels shape: (200, 2)\n"
     ]
    }
   ],
   "source": [
    "# [1, 1] quarter\n",
    "d1 = np.random.rand(50, 2)\n",
    "# [-1, -1] quarter\n",
    "d2 = (np.random.rand(50, 2) - 1)\n",
    "# Labels\n",
    "l1 = np.ones(len(d1) + len(d2))\n",
    "l1 = np.column_stack((l1, np.zeros(len(d1) + len(d2))))\n",
    "\n",
    "# [-1, 1] quarter\n",
    "d3 = np.random.rand(50, 2)\n",
    "d3[:,0] -= np.ones(50)\n",
    "# [1, -1] quarter\n",
    "d4 = np.random.rand(50, 2)\n",
    "d4[:,1] -= np.ones(50)\n",
    "# Labels\n",
    "l2 = np.zeros(len(d3) + len(d4))\n",
    "l2 = np.column_stack((l2, np.ones(len(d1) + len(d2))))\n",
    "\n",
    "# Concatenate the data in a 2D big tensor\n",
    "data = nets.Tensor(np.concatenate((d1, d2, d3, d4)))\n",
    "labels = nets.Tensor(np.concatenate((l1, l2)).astype(int))\n",
    "\n",
    "print(f\"data shape: {data.shape}\")\n",
    "print(f\"labels shape: {labels.shape}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy: 98.50%\n"
     ]
    }
   ],
   "source": [
    "# Gradients accumulates, therefore we need to set them to zero at each iteration\n",
    "model.zero_grad()\n",
    "# Predictions\n",
    "predictions = model(data)\n",
    "loss = criterion(predictions, labels)\n",
    "\n",
    "pred_classes = np.argmax(predictions.data, axis=1)\n",
    "labels = np.argmax(labels.data, axis=1)\n",
    "\n",
    "accuracy = (pred_classes == labels).astype(float).sum() / 200\n",
    "print(f'accuracy: {accuracy * 100:2.2f}%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5.6. Boundary decision"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sanity check\n",
      "data shape: (200, 2)\n",
      "labels shape: (200,)\n"
     ]
    }
   ],
   "source": [
    "print(\"Sanity check\")\n",
    "print(f\"data shape: {data.shape}\")\n",
    "print(f\"labels shape: {labels.shape}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(x):\n",
    "    # Convert into numpy element to Tensor\n",
    "    x = nets.Tensor(x)\n",
    "    # Predict and return the class index\n",
    "    pred = model.forward(x)\n",
    "    pred_classes = np.argmax(pred.data, axis=1)\n",
    "    return pred_classes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_decision_boundary(pred_func, X, y):\n",
    "    # Set min and max values for the bbox\n",
    "    x_min, x_max = X[:, 0].min(), X[:, 0].max()\n",
    "    y_min, y_max = X[:, 1].min(), X[:, 1].max()\n",
    "    h = 0.01\n",
    "    # Generate a grid of points with distance h between them\n",
    "    xx, yy=np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))\n",
    "    # Predict the function value for the whole gid\n",
    "    Z = pred_func(np.c_[xx.ravel(), yy.ravel()])\n",
    "    Z = Z.reshape(xx.shape)\n",
    "    # Plot the contour and training examples\n",
    "    plt.subplots()\n",
    "    plt.contourf(xx, yy, Z, alpha=0.3, cmap='coolwarm')\n",
    "    scatter = plt.scatter(X[:, 0], X[:, 1], c=y, cmap='coolwarm')\n",
    "    plt.legend(*scatter.legend_elements(), loc=1, frameon=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 72372 characters]\n",
      "text/plain": [
       "<Figure size 720x576 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_decision_boundary(lambda x : predict(x), data.data, labels)\n",
    "\n",
    "# Save the figure\n",
    "# plt.tight_layout()\n",
    "# plt.savefig(\"img/xor_classification.png\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " We have seen the main functionalities of **NETS** and how to set up a model and use one.\n",
    " \n",
    " There will be more tutorials on how to build such library from scratch, and other use cases with **CNN** and **RNN**."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
