week-4-basic-model.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get all the variables, classes and functions we defined in the previous lessons\n",
    "from vars.week3 import *\n",
    "\n",
    "import torch.nn as nn\n",
    "from torchsummary import summary"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4. Defining simple models\n",
    "## 4.1 nn.Sequential\n",
    "The `Sequential` class is very simple: it accepts a sequence of neural network `Modules` as arguments and arranges them such that the output of one is automatically sent to the input of the next in line. This saves us a bit of time writing some code, but has some drawbacks, as we shall see shortly. The following is the simplest possible neural network. It consists only of an input layer, 1 hidden layer and an output layer. It is good practice to print out the summary of your network using `torchsummary.summary`. This lets you inspect your networks parameters and the input/output sizes of each layer. Interestingly, it also acts as a sort of sanity checker for your model, because it will complain if the input/output sizes of your layers aren't compatible with each other."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "----------------------------------------------------------------\n",
      "        Layer (type)               Output Shape         Param #\n",
      "================================================================\n",
      "           Flatten-1                  [-1, 784]               0\n",
      "            Linear-2                  [-1, 128]         100,480\n",
      "              ReLU-3                  [-1, 128]               0\n",
      "            Linear-4                   [-1, 10]           1,290\n",
      "================================================================\n",
      "Total params: 101,770\n",
      "Trainable params: 101,770\n",
      "Non-trainable params: 0\n",
      "----------------------------------------------------------------\n",
      "Input size (MB): 0.00\n",
      "Forward/backward pass size (MB): 0.01\n",
      "Params size (MB): 0.39\n",
      "Estimated Total Size (MB): 0.40\n",
      "----------------------------------------------------------------\n"
     ]
    }
   ],
   "source": [
    "\n",
    "import torch.nn as nn \n",
    "\n",
    "def get_simple_linear_net():\n",
    "    return nn.Sequential(\n",
    "        nn.Flatten(),\n",
    "        nn.Linear(28 * 28, 128),\n",
    "        nn.ReLU(),\n",
    "        nn.Linear(128, 10)\n",
    "    )\n",
    "\n",
    "from torchsummary import summary\n",
    "summary(get_simple_linear_net(), input_size=(1, 28, 28), device=\"cpu\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4.1.1 Simple training loop\n",
    "Now that we've defined a network, we can start training it! Let's define the simplest possible training function, for which we only require the model, number of training epochs, the dataloader and an optimisation function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def train_model(model, epochs, train_dl, optimiser):\n",
    "    msg = \"\"\n",
    "    model.train() \n",
    "    for epoch in range(epochs):\n",
    "        total_steps = len(train_dl)\n",
    "        correct = 0\n",
    "        total = 0\n",
    "\n",
    "        for batch_num, (image_batch, label_batch) in enumerate(train_dl):\n",
    "            # Prepare the data and label batches\n",
    "            batch_sz = len(image_batch)\n",
    "            output = model(image_batch)\n",
    "\n",
    "            losses = nn.CrossEntropyLoss()(output, label_batch)\n",
    "\n",
    "            #Zero gradients and backpropagate losses\n",
    "            optimiser.zero_grad()\n",
    "            losses.backwards()\n",
    "            optimiser.step() # updates model weights based on gradients\n",
    "\n",
    "            preds = torch.argmax(output, dim=1)\n",
    "            correct += int(torch.eq(preds, label_batch).sum())\n",
    "            total += batch_sz\n",
    "            minibatch_accuracy = 100 * correct / total\n",
    "\n",
    "            #### Fancy printing stuff, you can ignore this! ######\n",
    "            if (batch_num + 1) % 5 == 0:\n",
    "                print(\" \" * len(msg), end='\\r')\n",
    "                msg = f'Train epoch[{epoch+1}/{epochs}], MiniBatch[{batch_num + 1}/{total_steps}], Loss: {losses.item():.5f}, Acc: {minibatch_accuracy:.5f}'\n",
    "                print (msg, end='\\r' if epoch < epochs else \"\\n\",flush=True)\n",
    "            #### Fancy printing stuff, you can ignore this! ######"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from torch.optim import SGD\n",
    "\n",
    "# Defining network hyperparameters\n",
    "epochs = 5\n",
    "batch_sz = 32\n",
    "learning_rate = 0.005\n",
    "\n",
    "# Get train, validation and test dataloaders from the fucntion we \n",
    "# defined last week \n",
    "train_dl, val_dl, test_dl = ...\n",
    "\n",
    "# Create an instance of our network\n",
    "network = ...                    \n",
    "optim = SGD(network.parameters(), lr=learning_rate)  # Stochastic gradient descent optimiser\n",
    "# Call the training function on the network and use the hyperparameters\n",
    "# defined above\n",
    "..."
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4.1.2 Debrief: Simple model with simple training loop\n",
    "At the end of the training loop, our model performs pretty well - should be around 80-90% accuracy most of the time. This is definitely better than random chance, so our model seems to have learned something about the dataset and can make good predictions. But it could be better! Before we look into improving this, there is something else that needs fixing...\n",
    "\n",
    "# 4.1.3 Training device\n",
    "Something you may have noticed so far is that the training loop runs quite slowly. 5 epochs is not a very long time at all in the machine learning world and it still takes a while to complete. This because we've been asking the CPU to do all the tensor calculations needed to update the weights. This is a bad idea because GPUs are much more efficient at processing large amounts of data in parallel. You should always use a GPU to train machine learning models if one is available. Pytorch makes it very easy to detect GPU availability and transfer code you've written for a CPU to GPU:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DEVICE = ...\n",
    "\n",
    "def train_model_gpu(model, epochs, train_dl, optimiser):\n",
    "    msg = \"\"\n",
    "    for epoch in range(epochs):\n",
    "        total_steps = len(train_dl)\n",
    "        correct = 0\n",
    "        total = 0\n",
    "\n",
    "        model.train()\n",
    "        for batch_num, (image_batch, label_batch) in enumerate(train_dl):\n",
    "            batch_sz = len(image_batch)\n",
    "            \n",
    "            # Transferring image and label tensors to GPU #\n",
    "           ###############################################\n",
    "            \n",
    "            output = model(image_batch)\n",
    "            losses = nn.CrossEntropyLoss()(output, label_batch)\n",
    "                        \n",
    "            optimiser.zero_grad()\n",
    "            losses.backward()\n",
    "            optimiser.step()  \n",
    "            \n",
    "            preds = torch.argmax(output, dim=1)\n",
    "            correct += int(torch.eq(preds, label_batch).sum())\n",
    "            total += batch_sz\n",
    "            minibatch_accuracy = 100 * correct / total\n",
    "\n",
    "            #### Fancy printing stuff, you can ignore this! ######\n",
    "            if (batch_num + 1) % 5 == 0:\n",
    "                print(\" \" * len(msg), end='\\r')\n",
    "                msg = f'Train epoch[{epoch+1}/{epochs}], MiniBatch[{batch_num + 1}/{total_steps}], Loss: {losses.item():.5f}, Acc: {minibatch_accuracy:.5f}'\n",
    "                print (msg, end='\\r' if epoch < epochs else \"\\n\",flush=True)\n",
    "            #### Fancy printing stuff, you can ignore this! ######"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Finally, we need to transfer our model to the device as well, and can begin training\n",
    "\n",
    "# Instantiate simple network\n",
    "network = ...\n",
    "# Instatiate SGD optimiser using network params\n",
    "optim = ...\n",
    "# Transfer network to GPU just like we did the tensors earlier\n",
    "network = ...\n",
    "# Call the new training function\n",
    "...\n",
    "\n",
    "# You should see a speedup in training speed!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "sem-deep-learning",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}