{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00000-6ac93d33-7180-4000-8c7f-2b27aef6eacd",
    "deepnote_cell_type": "markdown"
   },
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00001-ed0a2f06-a5ae-4e02-8b83-0f52f76a92f1",
    "deepnote_cell_type": "markdown"
   },
   "source": [
    "Every science and engineering discipline relies on differentiation in some capacity, whether seeking to optimize system operations, deriving rates of change, or evaluating complex expressions. In this era of abundant computationally intensive tasks, evaluating gradients of any function (regardless of form) is both practical and valuable. The FADiff package addresses this task by automatically differentiating functions in either forward or reverse mode. By implementing automatic differentiation (AD), which sequentially evaluates elementary functions, FADiff avoids the complexity of symbolic differentiation and the precision issues of numerical differentiation. Additional information on implementation is below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00002-23402e9e-6c54-4fd5-8a52-e3294fe77c87",
    "deepnote_cell_type": "markdown"
   },
   "source": [
    "## Background"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00003-6a571975-82fb-4224-a696-3d6ed4f631a8",
    "deepnote_cell_type": "markdown"
   },
   "source": [
    "Automatic Differentiation (AD) is a set of techniques for evaluating derivatives precisely based on computation graphs, chain rules and other symbolic rules. Compared with manual calculation or a symbolic approach to calculating derivatives, it is highly convenient and fast since it frees users from tedius calculations and proofs. Compared with finite approximation (a.k.a numerical differentiation), it is more accurate in that it avoids truncation errors or round-off errors that might arise in symbolic differentiation when selecting a huge step (h) or a tiny step (h). (We've analyzed this point in HW4.). Due to these advantages, it has been widely used in scientific computing, machine learning, deep learning, etc. \n",
    "\n",
    "The mathematical background knowledge mainly includes matrix-vector products, the Jacobian matrix, the algegra of dual numbers, Taylor series expansions, higher-order derivatives, etc. We will discuss them in more details later. There are 2 evaluation modes in AD, **forward mode** and **reverse mode**.\n",
    "\n",
    "1. Forward mode performs the operation of evaluating the numerical derivative concurrently with evaluating the function itself on a computational graph.\n",
    "\n",
    "2. Reverse mode is an alternative to the forward mode. It uses the computation graph in forward mode to calculate the final output and then traveres reversely to perform the operation of evaluating derivatives. This mode is commonly used in deep learning and neural networks, in which it is also called backpropogation. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00004-5db04178-d588-410c-8783-67c6ad616879",
    "deepnote_cell_type": "markdown"
   },
   "source": [
    "### 1. Matrix-vector Products\n",
    "##### 1.1 Definition \n",
    "   Given an $m\\times n$  matrix $A_{m\\times n}$ and a vector $x\\in R^{n}$, there is a way to create a linear combination\n",
    "   \n",
    "$$\n",
    "x_1a_1 + x_2a_2 + ... + x_na_n \\in R^m \n",
    "$$\n",
    "\n",
    "using the columns $a_1, . . . , a_n$ of $A$, where $x=\\left[x_1,x_2,...,x_n \\right]^{T}$.\n",
    "\n",
    "##### 1.2 Notes\n",
    "1. Matrix-vector products are only valid when the sizes of the matrix and vector are compatible – the number of elements of vector $x$ must equal the number of columns of matrix $A$. The number of elements in the output vector must equal the number of rows in matrix $A$.\n",
    "2. We can interpret matrix-vector products as creating a linear transformation or a **map** from $R^n$ to $R^m$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00005-c19b5860-9b26-4150-b5fc-b37d5a98eaae",
    "deepnote_cell_type": "markdown"
   },
   "source": [
    "### 2. Two Evaluation Modes: Forward & Reverse, Jacobian Matrix.\n",
    "Automatic Differentiation (AD) can be applied on both scalar functions with one variable or functions with multiple variables. The derivative calculation of a single variable is relatively straightforward, while in situations with multiple variables, we will introduce a terminology called the Jacobian Matrix ($J$).\n",
    "\n",
    "\n",
    "Let's start from a general case with $x$ is a vector."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00006-0d13ad32-d233-4e83-b04c-b702681e735b",
    "deepnote_cell_type": "markdown"
   },
   "source": [
    "##### 2.1 Jacobian Matrix\n",
    "If $f$ is a matrix of multiple functions with multiple input variables, then denote $f$ as \n",
    "\n",
    "$$\n",
    "f=\\begin{bmatrix} f_1(x,y) \\\\ f_2(x,y) \\end{bmatrix}\n",
    "$$\n",
    "\n",
    "Then, the derivative of matrix $f$ is called the Jacobian Matrix $J$:\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "  J = \n",
    "  \\begin{bmatrix}\n",
    "    \\partial f_{1} / \\partial x & \\partial f_{1} / \\partial y \\\\\n",
    "    \\partial f_{2} / \\partial x & \\partial f_{2} / \\partial y\n",
    "  \\end{bmatrix}\n",
    "\\end{aligned}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00007-9d2f9f92-aa61-4684-b53e-c75e364f3a76",
    "deepnote_cell_type": "markdown"
   },
   "source": [
    "##### 2.2 Forward Mode\n",
    "A program can be written as a combination of several functions: $f = f_1 ... f_n$. Let's set $x_0$ as a vector in $R^n$ and $x_n$ as the output vector; then each $f_i$ is the transaction function (a generalized \"matrix\" from the definition of matrix-vector products), and \n",
    "\n",
    "$$\n",
    "x_1 = f_1x_0\n",
    "$$\n",
    "$$\n",
    "x_2 = f_2x_1\n",
    "$$\n",
    "$$\n",
    "...\n",
    "$$\n",
    "$$\n",
    "x_n=f_nx_{n-1}.\n",
    "$$\n",
    "From the chain rule, we have:\n",
    "\n",
    "$$\n",
    "\\dot{x_1} =  (J f_1 x_0)\n",
    "$$\n",
    "$$\n",
    "\\dot{x_2} =  (J f_2 x_1) \\times \\dot{x_1}\n",
    "$$\n",
    "$$\n",
    " ... \n",
    " $$\n",
    "$$ \n",
    "\\dot{x_n} = (J f_n x_{n-1})\\times \\dot{x_{n-1}}. \n",
    "$$\n",
    "\n",
    "The above process of evaluating derivatives is called **forward mode Automatic Differentiation**.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00008-00086732-90a9-495c-a984-3f19577fcfa7",
    "deepnote_cell_type": "markdown"
   },
   "source": [
    "##### 2.3 Reverse Mode\n",
    "If we take the transpose of both the left and right sides of equation (1),(2)...(n) above, then \n",
    "\n",
    "$${x_1}^\\prime = (f_1x_0)^T$$\n",
    "$${x_2}^\\prime = (f_2x_1)^T$$\n",
    "$$...$$\n",
    "$${x_n}^\\prime = (f_nx_{n-1})^T.$$\n",
    "\n",
    "From the chain rule, we have:\n",
    "\n",
    "$$ {x_{n-1}}^\\prime =  (J f_n x_{n-1})^T$$\n",
    "$$ {x_{n-2}}^\\prime =  (J f_{n-1} x_{n-2}) \\times {x_{n-1}}^\\prime $$\n",
    "$$ ... $$\n",
    "$$ {x_0}^\\prime = (J f_1 x_0)\\times {x_1}^\\prime. $$\n",
    "\n",
    "The above process of evaluating derivatives is called **reverse mode Automatic Differentiation**.\n",
    "\n",
    "\n"
   ]
  },
  {
   "attachments": {
    "image.png": {
     "image/png": "[encoded data, 2842284 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 2.4 Example of computational graph, forward and reverse mode.\n",
    "\n",
    "![image.png](attachment:image.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00010-1ab825af-5c90-4cc1-89de-d18db6fc648e",
    "deepnote_cell_type": "markdown"
   },
   "source": [
    "##### 2.5 When to use reverse or forward mode?\n",
    "\n",
    "The difference between forward and reverse mode lies in the start point of matrix multiplication.\n",
    "\n",
    "When the dimension of the input is less than that of the output, forward mode has fewer multiplication operations than reverse mode; comparably, when the dimension of the input is more than that of the output, reverse mode has fewer multiplication operations. \n",
    "\n",
    "Therefore, when the dimension of the input is less than that of the output, forward mode is more efficient; when the dimension of the input is more than that of the output, reverse mode is more efficient. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00011-359781dc-757d-4501-98ce-46f0bf896ad7",
    "deepnote_cell_type": "markdown"
   },
   "source": [
    "### 3. The algebra of dual numbers\n",
    "##### 3.1 Definition\n",
    "A dual number (z) is composed of a real part (a) and a dual part (b).  We denote it as \n",
    "\n",
    "$$z = a + \\epsilon b.$$\n",
    "\n",
    "##### 3.3 What's the effect of dual numbers on derivatives? \n",
    "The usage of a dual number augments the arithmetic in real number space to any input and allows the user to get the derivatives without calculating them. A function f(x) where x is a dual number can be re-written in a dual number format, where the real component is the function and the dual component contains the derivative (as we discussed in lecture 10).\n",
    "\n",
    "Generally, let $\\hat f$ denote the expansion of real-value function $f$ to dual number space, then\n",
    "\n",
    "$$ \\hat f(x_1+x_1^\\prime\\epsilon, ..., x_n + x_n^\\prime\\epsilon):=f(x_1,...,x_n)+\\dot f(x_1,...x_n) \\left(\\begin{array}{c}\n",
    "    x_1^\\prime\\\\ \n",
    "    .\\\\\n",
    "    .\\\\\n",
    "    .\\\\\n",
    "    x_n^\\prime\\\\\n",
    "  \\end{array}\\right)\\epsilon $$\n",
    "\n",
    "If $f$ is a matrix of multiple differentiable functions, then we can extend the above framework by replacing $\\dot f(x_1,...x_n)$ with $J f (x_1,...,x_n)$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00012-e34784ab-8ccf-42bb-9b39-60d417ca1521",
    "deepnote_cell_type": "markdown"
   },
   "source": [
    "### 4. Elemental functions\n",
    "\n",
    "Automatic Differentiation relies on the fact that we already know the derivative at each step. So, we need some elemental functions. A function is called \"elemental\" if it always returns the same result for the same argument values and has no side effect like modifying a global variable, etc. The result of calling an elemental function is the exact return value. \n",
    "\n",
    "Some examples are pow(), mean(), sqrt(), while printf(), rand() and time() are not elemental functions.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00013-7e1db5fe-fdad-4060-a893-2c835adafd7a",
    "deepnote_cell_type": "markdown"
   },
   "source": [
    "## How to use FADIFF\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00014-fd350bd6-9b0a-4418-a961-451ba7f873f1",
    "deepnote_cell_type": "markdown"
   },
   "source": [
    "<font color = 'red'>\n",
    "    \n",
    "    UPDATE\n",
    "\n",
    "We expect the use of our package, `FADiff`, to be largely through its API. The API primarily permits the use of objects and functions from NumPy. However, for certain areas of our implementation, our package requires the exclusive use of internally defined objects and functions. For example, we prohibit users from using external libraries for elementary functions (e.g., sine and cos) with variables and only allow them to use our package’s implementations for such functions. This helps reduce the potential for issues further on in the development process such as disuse or misuse of our package’s operator-overloaded functions, among other things. We will be clear in our documentation on how the user should use our package’s API including the proper use of variables and methods.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00015-b2245acc-d1d9-4f25-bace-5aef24e94592",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "### Run\n",
    "\n",
    "<font color = 'red'>\n",
    "    \n",
    "    UPDATE EXAMPLE CODE\n",
    "\n",
    "Our package can be downloaded from:\n",
    "<https://github.com/teamxvii/cs107-FinalProject> \n",
    "(make sure you are signed in to your GitHub account where you are a collaborator for our project). There you will see a green button called 'Code' and a button further to the left of that which should say 'master' (if not click on it to switch it to the 'master' branch). Click on the 'Code' button and then click on 'Download ZIP' in the window that pops up. This will download the package to your computer. Unzip its contents to the location of your code that will be using the package. For importability, rename the unzipped folder to something that does not contain hyphens. From here on out, it will be referred to as `cs107_FinalProject`. In your code file, only one header should be needed to import and use our entire package such as `from cs107_FinalProject.code.FADiff import *`. In addition, we implemented our package with Python 3.8.2 on Linux but other versions of Python may still be compatible. To retrieve the dependencies used in our package, navigate to the `cs107_FinalProject` folder in a terminal window and run the following:\n",
    "\n",
    "For Python 2 --\n",
    "```\n",
    "pip install -r requirements.txt\n",
    "```\n",
    "For Python 3 --\n",
    "```\n",
    "pip3 install -r requirements.txt\n",
    "```\n",
    "\n",
    "An example demonstrating the use of our package is shown below (the following code should be in your code file):\n",
    "\n",
    "```\n",
    "from cs107_FinalProject.code.FADiff import *      # Imports our package\n",
    "\n",
    "x = FADiff(5, 1)      # Creates a variable x with value of 5 and derivative of 1\n",
    "f = x + 2             # Creates a function f using variable x above\n",
    "print(f.val)          # Prints value of f\n",
    "print(f.der)          # Prints derivative of f\n",
    "```\n",
    "\n",
    "To run the above code, in a terminal window, navigate to the folder that contains the `cs107_FinalProject` folder and your code file and run the following:\n",
    "\n",
    "For Python 2 --\n",
    "```\n",
    "python your_code_file.py\n",
    "```\n",
    "For Python 3 --\n",
    "```\n",
    "python3 your_code_file.py\n",
    "```\n",
    "\n",
    "where 'your_code_file' is the name of your code file. The following output should then be rendered:\n",
    "\n",
    "```\n",
    "7\n",
    "1\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00016-189a523d-6586-47c7-825b-bec625389f54",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "### Test\n",
    "Navigate to the `cs107_FinalProject/code` folder in a terminal window and run the\n",
    "following:\n",
    "\n",
    "```\n",
    "pytest test_main.py\n",
    "```\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00015-f1214468-40c5-463f-a994-887828249efe",
    "deepnote_cell_type": "markdown"
   },
   "source": [
    "## Software Organization\n",
    "\n",
    "### 1. What will the directory structure look like?\n",
    "\n",
    "<font color = red> \n",
    "    \n",
    "    UPDATE WHEN FINALIZED\n",
    "\n",
    "Currently, our directory structure looks like the following:\n",
    "```\n",
    "cs107_FinalProject/\n",
    "    code/\n",
    "        FADiff.py\n",
    "        test_main.py\n",
    "    docs/\n",
    "        milestone1.ipynb\n",
    "        milestone2.ipynb\n",
    "        milestone2_progress.ipynb\n",
    "    requirements.txt\n",
    "    README.md\n",
    "```\n",
    "However, we anticipate the directory structure to eventually look something like:\n",
    "```\n",
    "cs107_FinalProject/\n",
    "    src/\n",
    "        FADiff/\n",
    "            FADiff.py\n",
    "    includes/\n",
    "    docs/\n",
    "        milestone1.ipynb\n",
    "        milestone2.ipynb\n",
    "        milestone2_progress.ipynb\n",
    "    tests/\n",
    "        test_main.py\n",
    "    examples/\n",
    "    requirements.txt\n",
    "    README.md\n",
    "    LISCENSE.md\n",
    "    .travis.yml\n",
    "    setup.py\n",
    "```\n",
    "    \n",
    "### 2. What modules do you plan on including? What is their basic functionality?\n",
    "Our FADiff package contains a module named `FADiff.py`. `FADiff.py` will\n",
    "contain our main automatic differentiation class `FADiff()` as well as\n",
    "elementary functions that are used to calculate the derivatives of\n",
    "all the elementary functions our package supports such as sine and cosine. Our package\n",
    "also contains a module named `test_main.py` which contains our test class used in our\n",
    "testing.\n",
    "We also used NumPy as an external dependency for our calculations. As explained in the\n",
    "“How to Use FADiff” section earlier in this document, our implementation may need to\n",
    "limit the use of external packages or only use them internally\n",
    "(i.e., hidden from the API) in certain areas moving forward.\n",
    "\n",
    "### 3. Where will your test suite live? Will you use TravisCI? CodeCov?\n",
    "Currently, our tests are in the `cs107_FinalProject/code` folder in `test_main.py`.\n",
    "As mentioned earlier, we eventually plan to have it live in the `/tests` directory of\n",
    "the project and we used pytest for testing. Please see the\n",
    "How to Use FADiff section earlier in this document for running tests.\n",
    "### 4. How will you distribute your package (e.g. PyPI)?\n",
    "We are aiming to distribute through PyPI if time permits. \n",
    "### 5. How will you package your software? Will you use a framework? If so, which one and why? If not, why not?\n",
    "We won't be packaging the software using any sort of framework. The code will be clonable and installable via GitHub and via PyPI if time permits.\n",
    "### 6. Other considerations?\n",
    "After finishing Homework 4 and some upcoming lectures on various software topics such as containers, we may consider revising our software organization later on."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00016-6429e878-746a-43e7-b72e-9564285c6a7c",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "## Implementation\n",
    "\n",
    "### 1. Classes\n",
    "Our code relies on 6 main classes:\n",
    "* FADiff: API for package\n",
    "* funcVect: Vector function class for forward and reverse mode\n",
    "* fad/Scal: Scalar variable class for forward mode\n",
    "* fad/Vect: Vector variable class for forward mode\n",
    "* rev/Scal: Scalar variable class for reverse mode\n",
    "* rev/Vect: Vector variable class for reverse mode\n",
    "\n",
    "### 2. Class Attributes and Methods\n",
    "Attributes and methods for each class are described below:\n",
    "* FADiff: \n",
    "  - Contains global lists of all instances of forward mode scalar variables (fad/Scal), forward mode vector variables (fad/Vect), reverse mode scalar variables (rev/Scal), and reverse mode vector variables (rev/Vect)\n",
    "  - Contains methods for setting forward/reverse mode, creating a new scalar or vector object, and creating a new vector function\n",
    "* Scal (forward mode):\n",
    "  - Contains attributes for the value of a scalar variable, partial derivative of the variable, and parents of the variable, where \"parents\" are the original inputs (not intermediate values)\n",
    "  - Contains dunder methods for basic operations (addition, subtraction, multiplication, etc.), comparison operators (equals/not equals), and standard getters and setters\n",
    "* Vect (forward mode):\n",
    "  - Contains attributes for the value of a vector variable, partial derivative of the variable, and parents of the variable, where \"parents\" are the original inputs (not intermediate values)\n",
    "  - Contains dunder methods for basic operations (addition, subtraction, multiplication, etc.), comparison operators (equals/not equals), and standard getters and setters\n",
    "* Scal (reverse mode):\n",
    "  - Contains attributes for the value of a scalar variable and the partial derivative of the variable\n",
    "  - Contains dunder methods for basic operations (addition, subtraction, multiplication, etc.), comparison operators (equals/not equals), and standard getters and setters, all with a slightly different implementations than for forward mode to accomodate backpropogation mechanics\n",
    "* Vect (reverse mode):\n",
    "  - Contains attributes for the value of a scalar variable and the partial derivative of the variable\n",
    "  - Contains dunder methods for basic operations (addition, subtraction, multiplication, etc.), comparison operators (equals/not equals), and standard getters and setters, all with a slightly different implementations than for forward mode to accomodate backpropogation mechanics\n",
    "\n",
    "### 3. Data Structures\n",
    "Values for scalar variables are stored as floats or ints. Values for vector variables are stored in NumPy arrays. The partial derivative of a forward mode scalar variable is stored as a dictionary where the keys are all instances of the Scal (for scalar) class and the values are the partial derivative of the particular variable with respect to each key. The same dictionary structure is used for the partial derivative of forward mode vector variables, but the keys are all instances of the Vect (for vector) class and the values are partial derivatives in the form of a NumPy array. For reverse mode, scalar and vector variables, a dictionary is also used to store partial derivatives, but the values are lists of tuples containing the parent variable and the partial derivatives of the parent with respect to the key (an instance of the Scal/Vect class for reverse mode). Vector functions are stored as lists. Lastly, all instances of Scal or Vect objects are stored in lists of the AutoDiff class. \n",
    "\n",
    "### 4. Custom Elementary Functions\n",
    "The `Elems.py` module contains custom functions for basic elementary operations using the classes described above.\n",
    "\n",
    "<font color='red'>\n",
    "    \n",
    "### 5. External Dependencies (? - Sphinx still true?)\n",
    "We rely on NumPy for array storage and mathematical operations on scalar and vector variables (trigonmetric functions, exponentiation, etc). We will eventually use its array and linear algebra functions as well. We are considering also using Sphinx for auto-rendering and organizing our documentation.\n",
    "\n",
    "</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cell_id": "00016-992c141d-2dc9-4765-8dd6-3e4610e49a06",
    "deepnote_cell_type": "markdown",
    "tags": []
   },
   "source": [
    "## Final Extension\n",
    "\n",
    "### 1. Reflection of Milestone 2\n",
    "##### Future Features in Milestone 2\n",
    "\n",
    "Things to impelement next\n",
    "\n",
    "In this Milestone, we treat Jacobian as a scalar and it can only handle the case of single function of single input. Moving forward, we want to generalize the package for broader use cases.\n",
    "\n",
    "1. We will make the forward mode automatic differentiation object be able to access Jacobian Matrix.\n",
    "2. We will make the object to be for calculating partial direvatives.The challenging part is how to handle the number of variables we have, e.g if we define a class to calculate derivatives for multivariable functions f(x,y,z) and f(x,y,z,m,n), which data structure should we use as the attribute of the class?\n",
    "If we use an array instead of a scalar as the attribute, how can we implement the differentiation in an array?\n",
    "3. We might also think about how to calculate differentiation for polynomial functions like f(x) = x + sin(x) + cos(x).\n",
    "The challenging part here is how to implement a dunder method to handle the order of add or substraction\n",
    "Presumably, we might need to change the classes or change the data structures or add new modules. We should consult TAs and Prof. Sondak for more instruction and insights on implementation.\n",
    "\n",
    "##### Feedback from Milestone 2 [3pts] (D Sondak) 1 pt\n",
    "\n",
    "The future feature should be presented in a clearer manner including concrete steps on how you will realize the implementation of this extension. You are already required to implement the full version of forward mode (multiple functions and multiple inputs) for this project. The example provided (f(x) = x + sin(x) + cos(x)) should already work with the current implementation.\n",
    "\n",
    "Please set up a meeting with Tosin to discuss potential extensions as soon as possible.\n",
    "\n",
    "##### Reflection\n",
    "In Milestone2, we misunderstood the requirement for future feature as what we would do towards the minimum requirement on Forward Mode for the final deliverable. Then we communicated with TA and instructor and clarified the expectations and the right direction. \n",
    "Our extension are implementing the reverse mode and allowing users the flexibility of chosing and switching between forward and reverse mode. \n",
    "\n",
    "### 2. Description of extension feature\n",
    "1. We implemented reverse mode which realizes the same functionality as forward mode in a separate module.\n",
    "2. We also created a efficient and easy-to-use setting to allow users switching between modes: The default setting is forward mode, \n",
    "but if one wants to use reverse mode, he can just put \"ad.set_mode('reverse')\" at the top of the code file to call our API functions of reverse mode. \n",
    "If he wants to switch back to forward mode, just replace the \"reverse\" with \"forward\" in \"ad.set_mode()\".\n",
    "\n",
    "\n",
    "### 3.Background of extension -- reverse mode\n",
    "\n",
    "In reverse mode, the algorith first go through the forward evaluation trace to get the values of the outcome the intermediate nodes, and then reverse back to get the gradients. In calculating the gradients, it starts with the final outcome, setting its derivative with respect to itself to 1, and then reversely computes the precursive variable's derivative with respect to its inputs until it reaches the rood variable.\n",
    "<br>\n",
    "<br> In mathemetical form: \n",
    "- If $z_1,,,z_m$ are scalar inputs:\n",
    "$$z_1 = f_1(z_0)$$\n",
    "$$z_2 = f_2(z_1)$$\n",
    "$$...$$\n",
    "$$f = z_m = f_m(z_{m-1})$$\n",
    "Then the derivative of $f$ with respect to $z_0$ :\n",
    "\\begin{equation}\n",
    "\\begin{split}\n",
    "\\frac{dz_m}{dz_0}& = \\frac{dz_m}{dz_{m-1}}\\frac{dz_{m-1}}{dz_{0}}\\\\\n",
    "& = \\frac{dz_m}{dz_{m-1}} \\frac{dz_{m-1}}{dz_{m-2}}\\frac{dz_{m-2}}{dz_0}\\\\\n",
    "&...\\\\\n",
    "&=\\frac{dz_m}{dz_{m-1}} \\frac{dz_{m-1}}{dz_{m-2}}...\\frac{dz_{1}}{dz_0}\n",
    "\\end{split}\n",
    "\\end{equation}\n",
    "\n",
    "- If the input or intermediate variable is a vector:\n",
    "\\begin{equation}\n",
    "\\begin{split}\n",
    "u_1 & = f(v)\\\\\n",
    "u_2 & = g(v)\\\\\n",
    "t & = h(u_1,u_2)\\\\\n",
    "\\frac{d_t}{d_v} & = \\sum_{i}\\frac{dt}{du_i} \\frac{du_i}{dv}\n",
    "\\end{split}\n",
    "\\end{equation}\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Broader Impact and Inclusivity Statement \n",
    "### Broader Impact\n",
    "Regarding the broader impact portion, try to think about the ways people will use or misuse your software. What are the consequences? How should people use it responsibly? Are there any ethical implications? The NeurIPS website has a number of references to get you started on thinking about this:\n",
    "\n",
    "1. On a positive side, our software gives users a flexible, convenient problem-solving tool in automatically calculating derivatives in multiple settings without coding each steps on their own or taking care of every step on a white board and iterate, especially in large-scale computation and complex networks.\n",
    "Considering a 20 layers neural network with 100 nodes on each layer, hand-coding the derivatives and back propogation is time-consuming and may cause bugs and errors in each steps easily. \n",
    "However, by using this package, the users just define their activation function, loss function, and the software will do all the operation automatically in a few seconds with respect to every parameter.\n",
    "Moreover, our work also makes it straight-forward to implement gradient-based learning algorithm like gradient descent optimization, etc, which would arouse oceans of implication in both scientific research and real-word application.\n",
    "\n",
    "2. However, there can also be some situations when people misuse our software. First, it only supports the minimum requirement and can't support the case: \"vector input with vector functoins\" at least at current stage. Second, it can only give us the first derivative. \n",
    "Due to these limitation, we have to call this out in the user manual in case the users implement our software in unsuitable cases or causing errors.\n",
    "\n",
    "3. We are algo taking the risk of passivating human intelligence of young students. Considerting a young students who are studying calculus and derivatives or maching learning theories. Presumably that it is good for them to learn every details using hands-on experience. However, what if he uses our software in exams, assignments or interviews for accurate answers without digging down to the every details and calculation.\n",
    "Although using software is fast and efficiency in a high priority in many cases, it is harmful for a beginner to take the short cut. This software would likely encourage them to cheat and omit the \"repetitive\" but helpful programming practice.\n",
    "\n",
    "4. Another ethical consideration is that we might taking the risk of causing negative downstream effects. We have no knowledge of what the exact purpose of using our software, whether it is legal or peril. If it is for illegal usage, creating product that might cause harmful downstream effect to the society, our software will be involved in this\n",
    "ethical concerns which is not what we are expecting. \n",
    "\n",
    "### Inclusivity Statement\n",
    "It has been a hot issue that computer science and software development  should encourage diversity and be more inclusive to minority groups, for example, women, under-represented groups, people with disability, etc.\n",
    "Our software should also serve as a practice to pomote diversity, equity and inclusion and enable a equal contribution to our code base for all groups and individuals. \n",
    "\n",
    "1. The first thing we would advocate is to make our software a open source to everyone, enabling them to create pull request and add any comment to the development and also make this conversation public. \n",
    "\n",
    "2. Second, we would also ask users not to include their identity like gender/race/education level/institutions when they post, raise errors or create pull request on the repo. In this way, \n",
    "we can encourage more contribution from minority groups (under-represented students) without biasing people behavior by letting them know that only \"professional masters\" can contribute.\n",
    "If possible, we hope to add multi-language translated user manual for users from different parts of the world in order to empowering a diversed participation.\n",
    "\n",
    "3. Finally, we would also provide guidance and resources to engage economically disadvantaged students and people with disabilities.\n",
    "And always put a statement on our manual or software page phrasing the diversity and inclusivity statement and we actively encourage contribution from people from any\n",
    "age, culture, ethnicity, gender, geographic location, race and economic background."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Future Work\n",
    "\n",
    "### What we want to add\n",
    "Our package only calculates the first-order derivative, in the future, we want to add higher-order derivatives. \n",
    "\n",
    "### Why\n",
    "Firs-order derivative can only tell us the direction of movement. But the higher order derivatives can enable more use cases, for example, the second derivative can tell the rate of increase or decrease; the third order derivative can be useful in analyzing the jerk or jolt over time, an important quantity in engineering and motion control. From a statistical prospective, higher-order derivatives enables Riemannian Hamiltonian Monte Carlo sampler to overcome severe geometrical pathologies that first-order sampling cannot by using the second and third order derivatives of the log posterior\n",
    "distribution.\n",
    "\n",
    "### How to implement\n",
    "Let's take second-order derivative as an example, we could add a feature to allow our current package to return the simbolic function of the first-order derivative, based on which, we can apply the first-order automatic differentiation twice: Applying the forward mode twice to get a Hessian matrix ($H = f^{''}$) (or Hessian vector product) or we calculate truncated Taylor series in a particular direction to get a higher-order derivative of scalar inputs. "
   ]
  }
 ],
 "metadata": {
  "deepnote_execution_queue": [],
  "deepnote_notebook_id": "53cfc77d-cb46-4fc0-893b-180cbb28d26a",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
