{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Every science and engineering discipline relies on differentiation in some capacity, whether seeking to optimize system operations, deriving rates of change, or evaluating complex expressions. In this era of abundant computationally intensive tasks, evaluating gradients of any function (regardless of form) is both practical and valuable. The FADiff package addresses this task by automatically differentiating functions using forward mode. By implementing automatic differentiation (AD), which sequentially evaluates elementary functions, FADiff avoids the complexity of symbolic differentiation and the precision issues of numerical differentiation. Additional information on implementation is below."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Background"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Automatic Differentiation (AD) is a set of techniques for evaluating derivatives precisely based on computation graphs, chain rules and other symbolic rules. Compared with manual calculation or symbolic approach to calculating derivatives, it is highly convenient and fast since it frees users from tedius calculation and proof. Compared with finite approximation (a.k.a numerical differentiation), it is more accurate in that it avoids truncation errors or rounding-off errors that might arouse in symbolic differentiation when selecting a huge step (h) or a tiny step (h). (We've analyzed this point in HW4.). Due to these advantages, it has been widely used in scientific computing, machine learning, deep learning, etc. \n",
    "\n",
    "The mathematical background knowledge mainly includes matrix-vector product, Jacobian matrix, the algegra of dual numbers, Taylor's series expansion, higher-order derivatives, etc. We will discuss them in more details later. There are 2 evaluation modes in AD, **forward mode** and **reverse mode**.\n",
    "\n",
    "1. Forward mode performs the operation of evaluating the numerical derivative concurrently with evaluating the function itself on a computational graph.\n",
    "\n",
    "2. Reverse mode is an alternative to the forward mode. It uses the computation graph in forward mode to calculate the final output and then traveres reversely to perform to operation of evaluating derivatives. This mode is commonly used in deep learning and neural networks, in which it is also called backpropogation. \n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Matrix-vector Products\n",
    "##### 1.1 Definition \n",
    "   Given an $m\\times n$  matrix $A_{m\\times n}$ and a vector $x\\in R^{n}$, there is a way to create a linear combination\n",
    "   $$x_1a_1 + x_2a_2 + ... + x_na_n \\in R^m $$ using the columns $a_1, . . . , a_n$ of $A$, where $x=\\left[x_1,x_2,...,x_n \\right]^{T}$.\n",
    "\n",
    "##### 1.2 Notes\n",
    "1. Matrix-vector products are only valid when the sizes of the matrix and vector are compatible – the number of elements of vector $x$ must equal the number of columns of matrix $A$. The number of elements in the output vector must equal to the number of rows in matrix $A$.\n",
    "2. We can interpret the matrix-vector products as creating a linear transformation or a **map** from $R^n$ to $R^m$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Two Evaluation Mode: Forward & Reverse, Jacobian Matrix.\n",
    "Automatic Differentiation (AD) can be applied on both scalar functions with one variable or functions with multiple variables. The derivative calculation of a single variable is super straight forward, while in the situations with multiple\n",
    "variables, we will introduce a terminology called Jacobian Matrix ($J$).\n",
    "\n",
    "\n",
    "Let's start from a general case with $x$ is a vector)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 2.1 Jacobian Matrix\n",
    "If f is a matrix of multiple functions with multiple input variables, then denote $f$ as \n",
    "$$f=\\begin{bmatrix} f_1(x,y) \\\\ f_2(x,y) \\end{bmatrix}$$\n",
    "Then, the derivative of matrix f is called Jacobian Matrix $J$:\n",
    "\\begin{align*}\n",
    "  J = \n",
    "  \\begin{bmatrix}\n",
    "    \\partial f_{1} / \\partial x & \\partial f_{1} / \\partial y \\\\\n",
    "    \\partial f_{2} / \\partial x & \\partial f_{2} / \\partial y\n",
    "  \\end{bmatrix}\n",
    "\\end{align*}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 2.2 Forward Mode\n",
    "A program can be written as a combination of several functions: $f = f_1 ◦ · · · ◦ f_n$, let's set x_0 is a vector in $R^n$, $x_n$ is the output vector, each $f_i$ is the transaction function (a generalized \"matrix\" from the definition of matrix-vector products), then \n",
    "$$x_1 = f_1x_0$$\n",
    "$$x_2 = f_2x_1$$\n",
    "$$...$$\n",
    "$$x_n=f_nx_{n-1}.$$\n",
    "From the chain rule, we have:\n",
    "$$ \\dot{x_1} =  (J f_1 x_0)\\tag{1}$$\n",
    "$$ \\dot{x_2} =  (J f_2 x_1) \\times \\dot{x_1}\\tag{2}$$\n",
    "$$ ... $$\n",
    "$$ \\dot{x_n} = (J f_n x_{n-1})\\times \\dot{x_{n-1}}\\tag{n}. $$\n",
    "\n",
    "The above process of evaluating derivatives is called **forward mode Automatic Differentiation**.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 2.3 Reverse Mode\n",
    "If we take transpose on both left and right sides of equation (1),(2)...(n) above, then \n",
    "$${x_1}^\\prime = (f_1x_0)^T$$\n",
    "$${x_2}^\\prime = (f_2x_1)^T$$\n",
    "$$...$$\n",
    "$${x_n}^\\prime = (f_nx_{n-1})^T.$$\n",
    "From the chain rule, we have:\n",
    "$$ {x_{n-1}}^\\prime =  (J f_n x_{n-1})^T$$\n",
    "$$ {x_{n-2}}^\\prime =  (J f_{n-1} x_{n-2}) \\times {x_{n-1}}^\\prime $$\n",
    "$$ ... $$\n",
    "$$ {x_0}^\\prime = (J f_1 x_0)\\times {x_1}^\\prime. $$\n",
    "\n",
    "The above process of evaluating derivatives is called **reverse mode Automatic Differentiation**.\n",
    "\n",
    "\n"
   ]
  },
  {
   "attachments": {
    "image.png": {
     "image/png": "[encoded data, 2842284 characters]"
    }
   },
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 2.4 Example of computational graph, forward and reverse mode.\n",
    "\n",
    "![image.png](attachment:image.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### 2.5 When to use reverse or forward mode?\n",
    "\n",
    "The difference between forward and reverse mode lies in the start point of matrix multiplication. \n",
    "From the view of the times of multiplication operation, when the dimension of input is less than that of the output, forward mode has less multiplication operations than reverse mode; comparably, when the dimension of input is more than that of the output, reverse mode has less multiplication operations. \n",
    "Therefore, when the dimension of input is less than that of the output, forward mode is more efficient; when the dimension of input is more than that of the output, reverse mode is more efficient. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. The algebra of dual number \n",
    "##### 3.1 Definition\n",
    "A dual number (z) is composed of a real part (a) and a dual part (b).  We denote it as $$z = a + \\epsilon b$$.\n",
    "\n",
    "##### 3.3 What's the effect of dual numbers on derivatives? \n",
    "The usage of dual number augments the arithmetic in real number space to any input and allows the user to get the derivatives without calculating them. A function f(x) where x is a dual number can be re-written in a dual number format, where the real component is the function and dual component contains the derivative (as we discussed in lecture 10).\n",
    "\n",
    "Generally, let $\\hat f$ denotes the expansion of real-value function $f$ to dual number space, then\n",
    "$$ \\hat f(x_1+x_1^\\prime\\epsilon, ..., x_n + x_n^\\prime\\epsilon):=f(x_1,...,x_n)+\\dot f(x_1,...x_n) \\left(\\begin{array}{c}\n",
    "    x_1^\\prime\\\\ \n",
    "    .\\\\\n",
    "    .\\\\\n",
    "    .\\\\\n",
    "    x_n^\\prime\\\\\n",
    "  \\end{array}\\right)\\epsilon $$\n",
    "\n",
    "If $f$ is a matrix of multiple differentiable functions, then we can extend the above framework by replacing $\\dot f(x_1,...x_n)$ with $J f (x_1,...,x_n)$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Elemental functions\n",
    "\n",
    "Automatic Differentiation relies on the fact that we've already know the derivative at each step. So, we need some elemental functions. A function is called elemental function if it always returns the same result for same argument values and it has no side effect like modifying a global variable, etc. The result of calling a elemental function is the exact return value. \n",
    "Some examples are pow(), mean(), sqrt(), while printf(), rand() and time() are not elemental functions.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How to use FADIFF\n"
   ]
  },
  {
   "source": [
    "We expect the use of our package, `FADiff`, to be largely through its API. Where necessary or practical, our API may permit the use of objects and functions from NumPy or other widely-used external libraries. However, for certain areas of our implementation, we expect our package to require the exclusive use of internally defined objects and functions. For example, we might prohibit users from using external libraries for elementary functions (e.g., sin and cos) and only allow them to use our package’s implementations for such functions. This may help to reduce the potential for issues further on in the development process such as disuse or misuse of our package’s operator-overloaded functions, among other things. We will be clear in our documentation on how the user should use our package’s API including the proper use of variables and methods. Nonetheless, only one header should be needed to import and use our entire package. An example demonstrating the use of our package in pseudocode is shown below:\n",
    "\n",
    "```\n",
    "from fadiff.FADiff import AutoDiff as AutoDiffClassAlias     # Imports our package\n",
    "\n",
    "ad = AutoDiffClassAlias()                    # Instantiate an AD class instance\n",
    "f_x = x - ad.exp(-2.0 * ad.sin(4.0 * x))     # Defines a function f(x)\n",
    "f_prime = ad.get_derivative(f, point)        # Calculates f’ at given point using AD\n",
    "sin_x = ad.sin(x)                            # Calculates sin of x\n",
    "sin_prime_x = ad.sin_dot(x)                  # Derives sin of x\n",
    "```\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "## Software Organization\n",
    "\n",
    "### 1. What will the directory structure look like?\n",
    "We anticipate the direcotry strucutre to look something like:\n",
    "```\n",
    "cs107project/\n",
    "    src/\n",
    "        FADiff/\n",
    "            FADiff.py\n",
    "            ElemDerivs.py\n",
    "    includes/\n",
    "    docs/\n",
    "    tests/\n",
    "    examples/\n",
    "    README.md\n",
    "    LISCENSE.md\n",
    "    .travis.yml\n",
    "    setup.py\n",
    "```\n",
    "### 2. What modules do you plan on including? What is their basic functionality?\n",
    "Our FADiff package will contain a module named `FADiff.py` and another called `ElemDerivs.py` (short for “elementary derivatives”). `FADiff.py` will contain our main automatic differentiation class and `ElemDerivs.py` will contain functions that calculate the derivatives of all the elementary functions our package supports such as sin and cos. `FADiff.py` will import `ElemDerivs.py` so that it can use its functions. We also plan to use external module dependencies like NumPy or similar for handling matrices, vectors, and other linear algebra data types as well as for our calculations. As explained in the “How to Use FADiff” section earlier in this document, our implementation may need to limit the use of external packages or only use them internally (i.e., hidden from the API) in certain areas.\n",
    "\n",
    "### 3. Where will your test suite live? Will you use TravisCI? CodeCov?\n",
    "We will use TravisCI to run tests which will live in the `/tests` directory of the project. \n",
    "### 4. How will you distribute your package (e.g. PyPI)?\n",
    "We are aiming to distribute through PyPI if time permits. \n",
    "### 5. How will you package your software? Will you use a framework? If so, which one and why? If not, why not?\n",
    "We won't be packaging the software using any sort of framework. The code will be clonable and installable via GitHub and via PyPI if time permits.\n",
    "### 6. Other considerations?\n",
    "After finishing Homework 4 and some upcoming lectures on various software topics such as containers, we may consider revising our software organization later on.\n",
    "\n",
    "## Implementation\n",
    "### 1. What are the core data structures?\n",
    "We plan to use a list as our main data structure to hold the nodes (intermediate steps) including input parameters and output functions of an evaluation trace. An element in the list would represent one node in a trace and may be a list itself or perhaps an instance of a class. It would also contain pertinent data for that particular node including what parent or child nodes it has if any. We are also considering using a tuple to hold data that are handled together in forward mode such as the value of an elementary function at an intermediate step and the value of its derivative. Other data structures we plan to use include a NumPy array, int, and float. Furthermore, we may discover uses for other data structures that could supplant or augment parts of our implementation later on such as a tree, hash map, or dictionary.\n",
    "\n",
    "### 2. What classes will you implement?\n",
    "We will implement an automatic differentiation class. We are also considering creating a class to represent a node in the evaluation trace of the forward mode if needed.\n",
    "\n",
    "### 3. What method and name attributes will your classes have?\n",
    "We are aiming to only (or mainly) utilize one automatic differentiation class tentatively called AutoDiff in our package. This class’ attributes and methods will resemble the following pseudocode:\n",
    "```  \n",
    "def __init__():\n",
    "    self.parameters = {‘seed’: None, ‘jacobian’: None, ‘functions’: None,\n",
    "                       ‘input_values’: None, ‘output_values’: None}\n",
    "def parse(function):\n",
    "def get_derivative(function, point):\n",
    "def dunder methods:\n",
    "def elementary function methods:\n",
    "def elementary function derivative methods:\n",
    "def getters and setters:\n",
    "```\n",
    "We plan to have a `parameters` instance attribute that is a dictionary for persisting values including our seed vector.\n",
    "The `parse` method parses the given function for use in our package’s calculations. This may not be a needed method as we are hoping that our operator-overloading functions can handle parsing for us by design. If we do find a need for it later on, we may actually put this functionality in a class of its own and define variables inside it and so forth.\n",
    "The `get_derivative` method returns the derivative of the given function at the given point. Our implementation will provide a default seed value if the user doesn’t include one in as an argument.\n",
    "The `dunder` methods will be used for operator overloading so that our package’s objects can work with mathematical and any needed operations.\n",
    "The `elementary function methods` will be used for all the elementary functions supported by our package.\n",
    "The `elementary function derivative methods` will be used to derive the elementary functions supported by our package (using ElemDerivs.py).\n",
    "The `getters and setters` will be useful for many things such as accessing and changing the seed value, or returning a particular node in the evaluation trace.\n",
    "We will also clarify with our client, David, what the requirements are for certain tasks such as how to handle the seed value.\n",
    "\n",
    "### 4. What external dependencies will you rely on?\n",
    "We will use NumPy for linear algebra data types like matrices and vectors as well as for our calculations. We are considering also using Sphinx for auto-rendering and organizing our documentation.\n",
    "\n",
    "### 5. How will you deal with elementary functions like sin, sqrt, log, and exp (and all the others)?\n",
    "Our implementation’s API would contain functions for evaluating all of the elementary functions our package supports which could involve operator overloading. Internally (i.e., hidden from the API), these functions will utilize NumPy or a similar external library for the evaluation of elementary functions. Additionally, we will create our own functions for calculating derivatives of elementary functions in our ElemDerivs.py module as explained earlier in this document. If a function is not supported by our API or implementation then our package won’t be able to recognize it and an error will be raised if such a function is used.\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "## Feedback\n",
    "\n",
    "### Milestone 1\n",
    "\n",
    "#### Comment\n",
    "\n",
    "Good job! Consider using a different data structure apart from list for your main data structure. You may want to use something a bit more robust and structured i.e. dictionary, tree, hashmap. Hopefully the upcoming homework will give you all a bit of insight into the functionalities/pros and cons of a data structure like a BST\n",
    "\n",
    "Oluwatosin Alliyu , Oct 27 at 9:27pm\n",
    "\n",
    "#### Group Response\n",
    "\n",
    "Per Oluwatosin’s feedback, we are considering a dictionary or a tree-like structure as our main data structure for our evaluation trace. Due to the binary nature of primitive operations (i.e., two inputs produce an output), we hope a tree could possibly be used in that respect. Each node in a trace will have an elementary function value and its derivative. If we use a dictionary it will allow us to add a key-value pair where the key is the name of the node and the value is a tuple in which the first element will be the elementary function’s value and the second, its derivative. We would greatly appreciate advice or direction as to whether a tree or a dictionary would best satisfy our use case.\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}