PyTorch 101 Part 1 - Computational Graphs and Autograd in PyTorch.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Computation Graphs \n",
    "\n",
    "When we design software to implement neural networks, we want to come up with a way that can allow us to seamlessly compute the gradients, regardless of the architecture type so that the programmer doesn't have to manually compute gradients when changes are made to the network. \n",
    "\n",
    "We represent the computation with a data structure called a **Computation graph**. A computation graph looks very similar to the diagram of the graph that we made in the image above. However, the nodes in a computation graph are basically operators. These operators are basically the mathematical operators except for one case, where we need to represent creation of a user-defined variable.\n",
    "\n",
    "![Computation Graph](images/computation_graph_forward.png)\n",
    "\n",
    "Notice that we have also denoted the leaf variables $ a, w_1, w_2, w_3, w_4 $ in the graph for sake of clarity. However, it should noted that they are not a part of the computation graph. What they represent in our graph is the special case for user-defined variables which we just covered as an exception.\n",
    "\n",
    "The variables, b,c and d are created as a result of mathematical operations, whereas variables a, w1, w2, w3 and w4 are initialised by the user itself. Since, they are not created by any mathematical operator, nodes corresponding to their creation is represented by their name itself. This is true for all the leaf nodes in the graph. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tensors: Basic Building Blocks of PyTorch\n",
    "\n",
    "Tensor is a data structure which is a fundamental building block of PyTorch. Tensors are pretty much like numpy arrays, except that unlike numpy, tensors are designed to take advantage of parallel computation capabilities of a GPU. A lot of Tensor syntax is similar to that of numpy arrays. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([[0.0000e+00, 0.0000e+00, 0.0000e+00, 0.0000e+00, 1.0785e-30],\n",
      "        [1.4013e-45, 3.2751e-35, 1.4013e-45, 1.0640e-30, 1.4013e-45],\n",
      "        [3.1121e-35, 1.4013e-45, 1.0996e-30, 1.4013e-45, 4.4889e-29]])\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "tsr = torch.Tensor(3,5)\n",
    "\n",
    "print(tsr)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One it's own, Tensor is just like a numpy ndarray. A data structure that can let you do fast linear algebra options. If you want PyTorch to create a graph corresponding to these operations, you will have to set the `requires_grad` attribute of the Tensor to True. \n",
    "\n",
    "The API can be a bit confusing here. There are multiple ways to initialise tensors in PyTorch. While some ways can let you explicitly define that the requires_grad in the constructor itself, others require you to set it manually after creation of the Tensor."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "t1 = torch.randn((3,3), requires_grad = True) \n",
    "\n",
    "t2 = torch.FloatTensor(3,3) # No way to specify requires_grad while initiating \n",
    "t2.requires_grad = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`requires_grad` is contagious. It means that when a Tensor is created by operating on other Tensors, the requires_grad of the resultant Tensor would be set True given at least one of the tensors used for creation has it's `requires_grad` set to `True`.\n",
    "\n",
    "Each Tensor has a something an attribute called `grad_fn`, which refers to the mathematical operator that create the variable. If `requires_grad` is set to False, `grad_fn` would be None. \n",
    "\n",
    "In our example where, `d = f(w_3b , w_4c)`, d's grad function would be the addition operator, since f adds it's to input together. Notice, addition operator is also the node in our graph that output's d. If our Tensor is a leaf node (initialised by the user), then the `grad_fn` is also None."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The grad fn for a is None\n",
      "The grad fn for d is <AddBackward0 object at 0x1062e4eb8>\n"
     ]
    }
   ],
   "source": [
    "import torch \n",
    "\n",
    "a = torch.randn((3,3), requires_grad = True)\n",
    "\n",
    "w1 = torch.randn((3,3), requires_grad = True)\n",
    "w2 = torch.randn((3,3), requires_grad = True)\n",
    "w3 = torch.randn((3,3), requires_grad = True)\n",
    "w4 = torch.randn((3,3), requires_grad = True)\n",
    "\n",
    "b = w1*a \n",
    "c = w2*a\n",
    "\n",
    "d = w3*b + w4*c \n",
    "\n",
    "L = 10 - d\n",
    "\n",
    "print(\"The grad fn for a is\", a.grad_fn)\n",
    "print(\"The grad fn for d is\", d.grad_fn)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Function\n",
    "\n",
    "All mathematical operations in PyTorch are implemented by the `torch.nn.Autograd.Function` class. This class has two important member functions we need to look at. \n",
    "1. The first is it's `forward`  function, which simply computes the output using it's inputs. \n",
    "2. The `backward` function takes the incoming gradient coming from the the part of the network in front of it. The gradient to be backpropagated from a function f is basically the gradient that is backpropagated to f from the layers in front of it multiplied by the local gradient of the output of f with respect to it's inputs. This is exactly what the `backward`\n",
    "function does. \n",
    "\n",
    "Let's again understand with our example of  $$ d = f(w_3b , w_4c) $$\n",
    "\n",
    "1. `d` is our Tensor here. It's grad_fn is ``<ThAddBackward>``. This is basically the addition operation since the function that creates `d` adds inputs.\n",
    "2. The `forward` function of the it's `grad_fn` receives the inputs $w_3b$ and $w_4c$ and adds them. This value is basically stored in the `d`\n",
    "3. The `backward` function of the `<ThAddBackward>` basically takes the the incoming gradient from the further layers as the input. This is basically $\\frac{\\partial{L}}{\\partial{d}}$ coming along the edge leading from `L` to `d`. This gradient is also the gradient of `L` w.r.t to `d` and is stored in `grad` attribute of the `d`. It can be accessed by calling `d.grad`.\n",
    "4. It then takes computes the local gradients  $\\frac{\\partial{d}}{\\partial{w_4c}}$ and $\\frac{\\partial{d}}{\\partial{w_3b}}$.\n",
    "5. Then the `backward` function multiplies the incoming gradient with the locally computed gradients respectively and \"sends\" the gradients to it's inputs by invoking the `backward` method of the `grad_fn` of their inputs. \n",
    "6. For example, the backward function of `<ThAddBackward>` associated with `d` invokes `backward` function of the `grad_fn` of the $w_4*c$ (Here, $w_4*c$ is a intermediate Tensor, and it's `grad_fn` is `<ThMulBackward>`.  At time of invocation of the `backward` function, the gradient $\\frac{\\partial{L}}{\\partial{d}} * \\frac{\\partial{d}}{\\partial{w_4c}} $ is passed as the input. \n",
    "7. Now, for the variable $w_4*c$, $\\frac{\\partial{L}}{\\partial{d}} * \\frac{\\partial{d}}{\\partial{w_4c}} $ becomes the incoming gradient, like $\\frac{\\partial{L}}{\\partial{d}} $ was for $d$ in step 3 and the process repeats. \n",
    "\n",
    "Flow of gradients is demonstrated below. \n",
    "\n",
    "![Backpropagated Gradients](images/full_graph.png)\n",
    "\n",
    "Algorithmically, here's how backpropagation happens with a computation graph. (Not the actual implementation, only representative)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "# def backward (incoming_gradients):\n",
    "# \tself.Tensor.grad = incoming_gradients\n",
    "\n",
    "# \tfor inp in self.inputs:\n",
    "# \t\tif inp.grad_fn is not None:\n",
    "# \t\t\tnew_incoming_gradients = //\n",
    "# \t\t\t  incoming_gradient * local_grad(self.Tensor, inp)\n",
    "\t\t\t\n",
    "# \t\t\tinp.grad_fn.backward(new_incoming_gradients)\n",
    "# \t\telse:\n",
    "# \t\t\tpass\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to compute derivatives in our neural network, we generally call `backward` on the Tensor representing our loss. Then, we backtrack through the graph starting from node representing the `grad_fn` of our loss.\n",
    "\n",
    "As described above, the `backward` function is recursively called through out the graph as we backtrack. Once, we reach a leaf node, since the `grad_fn` is None, but stop backtracking through that path.\n",
    "\n",
    "One thing to note here is that PyTorch gives an error if you call `backward` on vector-valued Tensor. This means you can only call `backward` on a scalar valued Tensor. In our example, if we assume a to be a vector valued Tensor, and call backward on L, it will throw up an error. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "grad can be implicitly created only for scalar outputs",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-9-513f7af8b861>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     15\u001b[0m \u001b[0mL\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;36m10\u001b[0m \u001b[0;34m-\u001b[0m \u001b[0md\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     16\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 17\u001b[0;31m \u001b[0mL\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/Users/ayooshmac/anaconda/lib/python3.5/site-packages/torch/tensor.py\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(self, gradient, retain_graph, create_graph)\u001b[0m\n\u001b[1;32m    100\u001b[0m                 \u001b[0mproducts\u001b[0m\u001b[0;34m.\u001b[0m \u001b[0mDefaults\u001b[0m \u001b[0mto\u001b[0m\u001b[0;31m \u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;31m`\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    101\u001b[0m         \"\"\"\n\u001b[0;32m--> 102\u001b[0;31m         \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mautograd\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbackward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgradient\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mretain_graph\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    103\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    104\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mregister_hook\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhook\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/ayooshmac/anaconda/lib/python3.5/site-packages/torch/autograd/__init__.py\u001b[0m in \u001b[0;36mbackward\u001b[0;34m(tensors, grad_tensors, retain_graph, create_graph, grad_variables)\u001b[0m\n\u001b[1;32m     82\u001b[0m         \u001b[0mgrad_tensors\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mlist\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mgrad_tensors\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     83\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 84\u001b[0;31m     \u001b[0mgrad_tensors\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_make_grads\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtensors\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mgrad_tensors\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     85\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mretain_graph\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     86\u001b[0m         \u001b[0mretain_graph\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcreate_graph\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Users/ayooshmac/anaconda/lib/python3.5/site-packages/torch/autograd/__init__.py\u001b[0m in \u001b[0;36m_make_grads\u001b[0;34m(outputs, grads)\u001b[0m\n\u001b[1;32m     26\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mout\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequires_grad\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     27\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0mout\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mnumel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 28\u001b[0;31m                     \u001b[0;32mraise\u001b[0m \u001b[0mRuntimeError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"grad can be implicitly created only for scalar outputs\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     29\u001b[0m                 \u001b[0mnew_grads\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mones_like\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mout\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     30\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mRuntimeError\u001b[0m: grad can be implicitly created only for scalar outputs"
     ]
    }
   ],
   "source": [
    "import torch \n",
    "\n",
    "a = torch.randn((3,3), requires_grad = True)\n",
    "\n",
    "w1 = torch.randn((3,3), requires_grad = True)\n",
    "w2 = torch.randn((3,3), requires_grad = True)\n",
    "w3 = torch.randn((3,3), requires_grad = True)\n",
    "w4 = torch.randn((3,3), requires_grad = True)\n",
    "\n",
    "b = w1*a \n",
    "c = w2*a\n",
    "\n",
    "d = w3*b + w4*c \n",
    "\n",
    "L = (10 - d)\n",
    "\n",
    "L.backward()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two ways to overcome this.\n",
    "\n",
    "1. If you just make a small change in the above code setting L to be the sum of all the errors, our problem will be solved\n",
    "\n",
    "2. Second way is, for some reason have to absolutely call backward on a vector function, you can pass a torch.ones of size of shape of the tensor you are trying to call backward with. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "\n",
    "a = torch.randn((3,3), requires_grad = True)\n",
    "\n",
    "w1 = torch.randn((3,3), requires_grad = True)\n",
    "w2 = torch.randn((3,3), requires_grad = True)\n",
    "w3 = torch.randn((3,3), requires_grad = True)\n",
    "w4 = torch.randn((3,3), requires_grad = True)\n",
    "\n",
    "b = w1*a \n",
    "c = w2*a\n",
    "\n",
    "d = w3*b + w4*c \n",
    "\n",
    "# Replace L = (10 - d) by \n",
    "L = (10 -d).sum()\n",
    "\n",
    "L.backward()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch \n",
    "\n",
    "a = torch.randn((3,3), requires_grad = True)\n",
    "\n",
    "w1 = torch.randn((3,3), requires_grad = True)\n",
    "w2 = torch.randn((3,3), requires_grad = True)\n",
    "w3 = torch.randn((3,3), requires_grad = True)\n",
    "w4 = torch.randn((3,3), requires_grad = True)\n",
    "\n",
    "b = w1*a \n",
    "c = w2*a\n",
    "\n",
    "d = w3*b + w4*c \n",
    "\n",
    "# Replace L = (10 - d) by \n",
    "L = (10 -d)\n",
    "\n",
    "L.backward(torch.ones(L.shape))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this way, we can have gradients for every Tensor , and we can update them using Optimisation algorithm of our choice. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "learning_rate = 0.5\n",
    "w1 = w1 - learning_rate * w1.grad\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda env:anaconda]",
   "language": "python",
   "name": "conda-env-anaconda-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}