01.getting-started-with-pytorch.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "01.getting-started-with-pytorch.ipynb",
      "provenance": [],
      "collapsed_sections": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T1bdIQ2NW1fr",
        "colab_type": "text"
      },
      "source": [
        "# Getting Started with PyTorch\n",
        "\n",
        "[PyTorch](https://pytorch.org/) is:\n",
        "\n",
        "> An open source machine learning framework that accelerates the path from research prototyping to production deployment.\n",
        "\n",
        "In my humble opinion, PyTorch is the [sweet](https://www.youtube.com/watch?v=-h4spfuMGDI) way to solve Machine Learning problems, in the real world! The vast community allows you to work state-of-the-art models and deploy them to production in no time (relatively speaking). Let's get started!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7Y-d-CKtEk72",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install -q -U torch watermark"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "E907oEHK8Evf",
        "colab_type": "code",
        "outputId": "5898f234-37d5-4136-b6df-c60ce0a0988f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 102
        }
      },
      "source": [
        "%load_ext watermark\n",
        "%watermark -v -p numpy,torch"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "CPython 3.6.9\n",
            "IPython 5.5.0\n",
            "\n",
            "numpy 1.17.5\n",
            "torch 1.4.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QKXpuuCt3rYB",
        "colab_type": "text"
      },
      "source": [
        "## PyTorch ❤ NumPy\n",
        "\n",
        "Do you know NumPy? If you do, learning PyTorch will be a breeze! If you don't, prepare to learn the skills that will guide you on your journey Machine Learning Mastery!\n",
        "\n",
        "Let's start with something simple:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wk173ueH4Uxy",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import torch\n",
        "import numpy as np"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vfzfCdLx5FEm",
        "colab_type": "code",
        "outputId": "7f9fe41b-e6a3-4941-9bb8-404c0f747ddf",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "a = np.array([1, 2])\n",
        "b = np.array([8, 9])\n",
        "\n",
        "c = a + b\n",
        "c"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([ 9, 11])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 4
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KwCAz4WM7XOy",
        "colab_type": "text"
      },
      "source": [
        "Adding the same arrays with PyTorch looks like this:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dDUOcIf07k8r",
        "colab_type": "code",
        "outputId": "2d574606-4f2d-4f94-8c7a-c06e55bc1e75",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "a = torch.tensor([1, 2])\n",
        "b = torch.tensor([8, 9])\n",
        "\n",
        "c = a + b\n",
        "c"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([ 9, 11])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "a-8tu_mlBVRx",
        "colab_type": "text"
      },
      "source": [
        "Fortunately, you can go from NumPy to PyTorch:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mYBBtOZ5BzZt",
        "colab_type": "code",
        "outputId": "0170ea79-c8f5-4070-8ed0-8f48792680b5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "a = torch.tensor([1, 2])\n",
        "\n",
        "a.numpy()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "array([1, 2])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n_tueio4B9UM",
        "colab_type": "text"
      },
      "source": [
        " and vice versa:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wbGxHHCmCI0y",
        "colab_type": "code",
        "outputId": "9bdee02c-655e-4af0-9ffb-00e1b751a7d9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "a = np.array([1, 2])\n",
        "torch.from_numpy(a)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([1, 2])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8eX91koTxYtc",
        "colab_type": "text"
      },
      "source": [
        "The good news is that the conversions incur almost no cost on the performance of your app. The NumPy and PyTorch store data in memory in the same way. That is, PyTorch is reusing the work done by NumPy.\n",
        "\n",
        "## Tensors\n",
        "\n",
        "Tensors are just n-dimensional number (including booleans) containers. You can find the complete list of supported data types at [PyTorch's Tensor Docs](https://pytorch.org/docs/stable/tensors.html).\n",
        "\n",
        "So, how can you create a Tensor (try to ignore that I've already shown you how to do it)?\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "YbBs1LnZ4In7",
        "colab_type": "code",
        "outputId": "22e79557-9936-423c-b3bd-b81872fe1127",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "torch.tensor([[1, 2], [2, 1]])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[1, 2],\n",
              "        [2, 1]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "qpfHgg9j40Zx",
        "colab_type": "text"
      },
      "source": [
        "You can create a tensor from floats:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "oQxFkDU94lZE",
        "colab_type": "code",
        "outputId": "5b7283e1-c4cf-42b8-d40c-c06fbc7208da",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "torch.FloatTensor([[1, 2], [2, 1]])"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[1., 2.],\n",
              "        [2., 1.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1_t2URO15H1m",
        "colab_type": "text"
      },
      "source": [
        "Or define the type like so:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V38yr_Jw47bz",
        "colab_type": "code",
        "outputId": "b4b41f5b-e62f-4423-8312-d5c5536b514b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "torch.tensor([[1, 2], [2, 1]], dtype=torch.bool)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[True, True],\n",
              "        [True, True]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JmRTLJtN58lV",
        "colab_type": "text"
      },
      "source": [
        "You can use a wide range of factory methods to create Tensors without manually specifying each number. For example, you can create a matrix with random numbers like this: "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IX7D1-fO59Tp",
        "colab_type": "code",
        "outputId": "46527178-2120-4d4a-b759-724f76193d08",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "torch.rand(3, 2)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[0.6686, 0.7622],\n",
              "        [0.0341, 0.5835],\n",
              "        [0.2423, 0.0651]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 11
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4vYL2FDC6emK",
        "colab_type": "text"
      },
      "source": [
        "Or one full of ones:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xpYH8oSK6haK",
        "colab_type": "code",
        "outputId": "329b7140-5f8b-4f7f-8eb5-41b8a1c4747e",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "torch.ones(3, 2)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[1., 1.],\n",
              "        [1., 1.],\n",
              "        [1., 1.]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "CK5KF34VGNeI",
        "colab_type": "text"
      },
      "source": [
        "PyTorch has a variety of useful operations:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NLMyFkALGd84",
        "colab_type": "code",
        "outputId": "d8498aef-e2c0-489f-db1d-7ea84fbd1f30",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "x = torch.tensor([[2, 3], [1, 2]])\n",
        "print(x)\n",
        "print(f'sum: {x.sum()}')"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "tensor([[2, 3],\n",
            "        [1, 2]])\n",
            "sum: 8\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QyzGvIsAIt-u",
        "colab_type": "text"
      },
      "source": [
        "Get the transpose of a 2-D tensor:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "rDgKzHgkIi4O",
        "colab_type": "code",
        "outputId": "4fe531d4-71fb-4b28-b455-6593d0e86b51",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "x.t()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[2, 1],\n",
              "        [3, 2]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 14
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d_JRK8brLZTh",
        "colab_type": "text"
      },
      "source": [
        "Get the shape of each dimension:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GvDZZ-igLaio",
        "colab_type": "code",
        "outputId": "04cff3fb-e665-485c-c155-2f562667dc48",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "x.size()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "torch.Size([2, 2])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zxjrhfUfR_rM",
        "colab_type": "text"
      },
      "source": [
        "Generally, performing some operation creates a new Tensor:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yy7LEtCCSszs",
        "colab_type": "code",
        "outputId": "9fba705c-10dd-4c87-a1ec-41f95346a4b0",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "y = torch.tensor([[2, 2], [5, 1]])\n",
        "z = x.add(y)\n",
        "z"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[4, 5],\n",
              "        [6, 3]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 16
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e9B-2vSaT0tx",
        "colab_type": "text"
      },
      "source": [
        "But you can do it in-place:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Fd2KmNhvT-pb",
        "colab_type": "code",
        "outputId": "663f18df-7f68-4936-c56f-2c22f5baf8e3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "x.add_(y)\n",
        "x"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[4, 5],\n",
              "        [6, 3]])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 17
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n1iRWWwOU_eq",
        "colab_type": "text"
      },
      "source": [
        "Almost all operations have an in-place version - the name of the operation, followed by an underscore."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "opIXKjEOxaas",
        "colab_type": "text"
      },
      "source": [
        "## Running on GPU\n",
        "\n",
        "At this point, you might be like: \"Why do I need PyTorch at all? All of this is perfectly doable with NumPy?\". PyTorch has three major superpowers: \n",
        "- you can run your operations on the GPU(s) (or something else)\n",
        "- [Autograd: automatic differentiation](https://pytorch.org/tutorials/beginner/blitz/autograd_tutorial.html)\n",
        "- A set of tools to build Neural Networks. Including several additional packages for working [with text](https://github.com/pytorch/text) or [images](https://github.com/pytorch/vision).\n",
        "\n",
        "Doing your Deep Learning computations on the GPU speeds up your experiment by a lot! And PyTorch makes it ridiculously easy to do it. Let's start by checking if GPU is available:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "8T3P8oGMZs03",
        "colab_type": "code",
        "outputId": "93191030-8161-41bb-d897-cc542809b018",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "device = torch.device(\"cuda\") if torch.cuda.is_available() else torch.device(\"cpu\")\n",
        "device"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "device(type='cuda')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 18
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0IELcj4DaEU9",
        "colab_type": "text"
      },
      "source": [
        "Good, we have a [CUDA](https://en.wikipedia.org/wiki/CUDA)-enabled GPU device on our hands. Let's store a Tensor on it:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ctE2IzK3bsBp",
        "colab_type": "code",
        "outputId": "a641a868-1124-46c3-da3b-22345bef28d4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "x = torch.tensor([[2, 3], [1, 2]])\n",
        "x.to(device)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[2, 3],\n",
              "        [1, 2]], device='cuda:0')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yZK72ECLb4fO",
        "colab_type": "text"
      },
      "source": [
        "Notice that our Tensor is now on device `cuda:0`. What can we do with it? Pretty much everything as before:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7pAshM7ncDBY",
        "colab_type": "code",
        "outputId": "36914743-b9ab-482c-840b-00c01cc46350",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "x = x.to(device)\n",
        "\n",
        "y = torch.tensor([[2, 2], [5, 1]])\n",
        "y = y.to(device)\n",
        "\n",
        "x.add(y)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "tensor([[4, 5],\n",
              "        [6, 3]], device='cuda:0')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 20
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BNjzckYYxiUq",
        "colab_type": "text"
      },
      "source": [
        "## Common Issues\n",
        "\n",
        "I got to be honest with you. You will fuck up, multiple times, before understanding how this whole thing works out. That's alright!\n",
        "\n",
        "However, there are a couple of things you can do that might minimize the frustrations along your journey:\n",
        "\n",
        "- Doing operations between GPU and CPU Tensors is **not allowed**\n",
        "- Size mismatch between Tensors occurs often and is (almost every time) easy to fix:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Xx8xt6SExcW0",
        "colab_type": "code",
        "outputId": "b29c3de8-3e9e-42b6-8d16-7ce93e03fbfc",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 217
        }
      },
      "source": [
        "a = torch.ones(2, 2) \n",
        "b = torch.ones(1, 3)\n",
        "a * b"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "error",
          "ename": "RuntimeError",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-21-b2a4e8765762>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0ma\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mones\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m2\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m2\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0mb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtorch\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mones\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m3\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0ma\u001b[0m \u001b[0;34m*\u001b[0m \u001b[0mb\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
            "\u001b[0;31mRuntimeError\u001b[0m: The size of tensor a (2) must match the size of tensor b (3) at non-singleton dimension 1"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d7_Aqx9mfCoV",
        "colab_type": "text"
      },
      "source": [
        "PyTorch is very descriptive in this case. When doing more complex stuff, you would want to check the shape of your Tensors obsessively, after every operation. Just print the size!\n",
        "\n",
        "- Running out of GPU memory: You might be leaking memory or too large of a dataset/model. Faster/better GPU always helps. But remember, you can solve really large problems with a single powerful GPU these days. Think carefully if that is not enough for you - why that is?\n",
        "\n",
        "## Conclusion\n",
        "\n",
        "Welcome to the dark side! You might've been working with Keras, TensorFlow, or another Deep Learning framework, until recently. Almost every framework is great, but PyTorch has really solid roots. Easy to use and understand, allows for fast experimentation and standard debugging tools apply! Enjoy!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wAt2CDnYjAc4",
        "colab_type": "text"
      },
      "source": [
        "## References\n",
        "\n",
        "- [\"PyTorch: A Modern Library for Machine Learning\" with Adam Paszke](https://www.youtube.com/watch?v=5bSAipCNqXo)\n",
        "- [Recitation 1 | Your First Deep Learning Code](https://www.youtube.com/watch?v=KrCp_yPVOxs)\n",
        "- [What is PyTorch?](https://pytorch.org/tutorials/beginner/blitz/tensor_tutorial.html)"
      ]
    }
  ]
}