01-tensor_tutorial.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# What is PyTorch?\n",
    "\n",
    "It’s a Python based scientific computing package targeted at two sets of audiences:\n",
    "\n",
    "-  Tensorial library that uses the power of GPUs\n",
    "-  A deep learning research platform that provides maximum flexibility and speed\n",
    "\n",
    "## Import the library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import torch  # <Ctrl> / <Shift> + <Return>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting help in Jupyter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.sq  # <Tab>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# What about all `*Tensor`s?\n",
    "# Press <esc> to get out of help\n",
    "torch.*Tensor?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.nn.Module()  # <Shift>+<Tab>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Annotate your functions / classes!\n",
    "torch.nn.Module?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "torch.nn.Module??"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dropping to Bash: magic!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# List all the files in the current directory\n",
    "!ls -lh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "# List all the files but with cleaner outputs for readability\n",
    "for f in $(ls *.*); do\n",
    "    echo $(wc -l $f)\n",
    "done"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Getting some general help\n",
    "%magic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Python native data types\n",
    "\n",
    "Python has many native datatypes. Here are the important ones:\n",
    "\n",
    " - **Booleans** are either `True` or `False`.\n",
    " - **Numbers** can be integers (1 and 2), floats (1.1 and 1.2), fractions (1/2 and 2/3), or even complex numbers.\n",
    " - **Strings** are sequences of Unicode characters, e.g. an html document.\n",
    " - **Lists** are ordered sequences of values.\n",
    " - **Tuples** are ordered, immutable sequences of values.\n",
    " - **Sets** are unordered bags of values.\n",
    " - **Dictionaries** are unordered bags of key-value pairs.\n",
    " \n",
    "See [here](http://www.diveintopython3.net/native-datatypes.html) for a complete overview.\n",
    "\n",
    "### More resources\n",
    "\n",
    " 1. Brief Python introduction [here](https://learnxinyminutes.com/docs/python3/).\n",
    " 2. Full Python tutorial [here](https://docs.python.org/3/tutorial/).\n",
    " 3. A Whirlwind Tour of Python [here](https://github.com/jakevdp/WhirlwindTourOfPython).\n",
    " 4. Python Data Science Handbook [here](https://github.com/jakevdp/PythonDataScienceHandbook)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Torch!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Generate a tensor of size 2x3x4\n",
    "t = torch.Tensor(2, 3, 4)\n",
    "type(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the size of the tensor\n",
    "t.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# t.size() is a classic tuple =>\n",
    "print('t size:', ' \\u00D7 '.join(map(str, t.size())))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# prints dimensional space and sub-dimensions\n",
    "print(f'point in a {t.numel()} dimensional space')\n",
    "print(f'organised in {t.dim()} sub-dimensions')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Mind the underscore!\n",
    "# Any operation that mutates a tensor in-place is post-fixed with an _.\n",
    "# For example: x.copy_(y), x.t_(), x.random_(n) will change x.\n",
    "t.random_(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This resizes the tensor permanently \n",
    "r = torch.Tensor(t)\n",
    "r.resize_(3, 8)\n",
    "r"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# As you can see zero_ would replace r with 0's which was originally filled with integers\n",
    "r.zero_()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This *is* important, sigh...\n",
    "s = r.clone()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# In-place fill of 1's\n",
    "s.fill_(1)\n",
    "s"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Because we cloned r, even though we did an in-place operation, this doesn't affect r\n",
    "r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Vectors (1D Tensors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creates a 1D tensor of integers 1 to 4\n",
    "v = torch.Tensor([1, 2, 3, 4])\n",
    "v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print number of dimensions (1D) and size of tensor\n",
    "print(f'dim: {v.dim()}, size: {v.size()[0]}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "w = torch.Tensor([1, 0, 2, 0])\n",
    "w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Element-wise multiplication\n",
    "v * w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Scalar product: 1*1 + 2*0 + 3*2 + 4*0\n",
    "v @ w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# In-place replacement of random number from 0 to 10\n",
    "x = torch.Tensor(5).random_(10)\n",
    "x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'first: {x[0]}, last: {x[-1]}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Extract sub-Tensor [from:to)\n",
    "x[1:2 + 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a tensor with integers ranging from 1 to 5, excluding 5\n",
    "v = torch.arange(1, 4 + 1)\n",
    "v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Square all elements in the tensor\n",
    "print(v.pow(2), v)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Matrices (2D Tensors)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a 2x4 tensor\n",
    "m = torch.Tensor([[2, 5, 3, 7],\n",
    "                  [4, 2, 1, 9]])\n",
    "m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m.dim()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(m.size(0), m.size(1), m.size(), sep=' -- ')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Returns the total number of elements, hence num-el (number of elements)\n",
    "m.numel()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Indexing row 0, column 2 (0-indexed)\n",
    "m[0][2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Indexing row 0, column 2 (0-indexed)\n",
    "m[0, 2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Indexing column 1, all rows (returns size 2)\n",
    "m[:, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Indexing column 1, all rows (returns size 2x2)\n",
    "m[:, [1]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Indexes row 0, all columns (returns 1x4)\n",
    "m[[0], :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Indexes row 0, all columns (returns size 4)\n",
    "m[0, :]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create tensor of numbers from 1 to 5 (excluding 5)\n",
    "v = torch.arange(1., 4 + 1)\n",
    "v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Scalar product\n",
    "m @ v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculated by 1*2 + 2*5 + 3*3 + 4*7\n",
    "m[[0], :] @ v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculated by \n",
    "m[[1], :] @ v"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add a random tensor of size 2x4 to m\n",
    "m + torch.rand(2, 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Subtract a random tensor of size 2x4 to m\n",
    "m - torch.rand(2, 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Multiply a random tensor of size 2x4 to m\n",
    "m * torch.rand(2, 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Divide m by a random tensor of size 2x4\n",
    "m / torch.rand(2, 4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m.size()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Transpose tensor m, which is essentially 2x4 to 4x2\n",
    "m.t()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Same as\n",
    "m.transpose(0, 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Constructors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create tensor from 3 to 8, with each having a space of 1\n",
    "torch.arange(3., 8 + 1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create tensor from 5.7 to -2.1 with each having a space of -3\n",
    "torch.arange(5.7, -2.1, -3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# returns a 1D tensor of steps equally spaced points between start=3, end=8 and steps=20\n",
    "torch.linspace(3, 8, 20).view(1, -1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a tensor filled with 0's\n",
    "torch.zeros(3, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a tensor filled with 1's\n",
    "torch.ones(3, 2, 5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a tensor with the diagonal filled with 1\n",
    "torch.eye(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Set default plots\n",
    "from plot_lib import set_default\n",
    "from matplotlib import pyplot as plt\n",
    "set_default()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Numpy bridge!\n",
    "plt.hist(torch.randn(1000).numpy(), 100);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.hist(torch.randn(10**6).numpy(), 100);  # how much does this chart weight?\n",
    "# use rasterized=True for SVG/EPS/PDF!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.hist(torch.rand(10**6).numpy(), 100);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Casting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Helper to get what kind of tensor types\n",
    "torch.*Tensor?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is basically a 64 bit float tensor\n",
    "m_double = m.double()\n",
    "m_double"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# This creates a tensor of type int8\n",
    "m_byte = m.byte()\n",
    "m_byte"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Move your tensor to GPU device 0 if there is one (first GPU in the system)\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "m.to(device)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Converts tensor to numpy array\n",
    "m_np = m.numpy()\n",
    "m_np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# In-place fill of column 0 and row 0 with value -1\n",
    "m_np[0, 0] = -1\n",
    "m_np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create a tensor of integers ranging from 0 to 4\n",
    "import numpy as np\n",
    "n_np = np.arange(5)\n",
    "n = torch.from_numpy(n_np)\n",
    "print(n_np, n)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# In-place multiplication of all elements by 2 for tensor n\n",
    "# Because n is essentiall n_np, not a clone, this affects n_np\n",
    "n.mul_(2)\n",
    "n_np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## More fun"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creates two tensor of size 1x4\n",
    "a = torch.Tensor([[1, 2, 3, 4]])\n",
    "b = torch.Tensor([[5, 6, 7, 8]])\n",
    "print(a.size(), b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Concatenate on axis 0, so you get 2x4\n",
    "torch.cat((a, b), 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Concatenate on axis 1, so you get 1x8\n",
    "torch.cat((a, b), 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Much more\n",
    "\n",
    "There's definitely much more, but this was the basics about `Tensor`s fun.\n",
    "\n",
    "*Torch* full API should be read at least once.\n",
    "Hence, go [here](http://pytorch.org/docs/0.3.0/torch.html).\n",
    "You'll find 100+ `Tensor` operations, including transposing, indexing, slicing, mathematical operations, linear algebra, random numbers, etc are described."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:dl-minicourse] *",
   "language": "python",
   "name": "conda-env-dl-minicourse-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}