training_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hello! Welcome to Carpl training workspace"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Contents\n",
    "\n",
    "1. [Background](#Background)\n",
    "1. [Requirements](#Requirements)\n",
    "1. [Setup](#Setup)\n",
    "1. [Data](#Data)\n",
    "1. [Train](#Train)\n",
    "1. [Host](#Host)\n",
    "\n",
    "---\n",
    "\n",
    "## Background\n",
    "In this training workspace you will get access to S3 bucket which contains the dataset and annotations you selected on Carpl platform. \n",
    "You can fetch that data in your current workspace to begin. We have provided template code to make your training experience better. \n",
    "\n",
    "\n",
    "/code folder contains script.py mnist.py<br>\n",
    "    script.py contains Network architecture, Data loaders, training code, testing code<br>\n",
    "\n",
    "\n",
    "It is also entrypoint for model deployment<br>\n",
    "1. input_fn : write custom input function to preprocess data from API \n",
    "2. model_fn : loads model into memory\n",
    "3. predict_fn : runs inferencing on resultant of input_fn\n",
    "4. output_fn  : write custom post processing to send data back to Carpl\n",
    "\n",
    "\n",
    "For more information about the PyTorch in SageMaker, please visit [sagemaker-pytorch-containers](https://github.com/aws/sagemaker-pytorch-containers) and [sagemaker-python-sdk](https://github.com/aws/sagemaker-python-sdk) github repositories.\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "![SAGE.drawio](SAGE.drawio.png)\n",
    "---\n",
    "\n",
    "## Setup\n",
    "\n",
    "_This notebook was created and tested on an ml.m4.xlarge notebook instance._\n",
    "\n",
    "Let's start by creating a SageMaker session and specifying:\n",
    "\n",
    "- The S3 bucket and prefix that you want to use for training and model data.  This should be within the same region as the Notebook Instance, training, and hosting.\n",
    "- The IAM role arn used to give training and hosting access to your data. See the documentation for how to create these.  Note, if more than one role is required for notebook instances, training, and/or hosting, please replace the `sagemaker.get_execution_role()` with a the appropriate full IAM role arn string(s).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Requirements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Install all your requirements for training model here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: sagemaker==2.110.0 in /opt/conda/lib/python3.7/site-packages (2.110.0)\n",
      "Requirement already satisfied: importlib-metadata<5.0,>=1.4.0 in /opt/conda/lib/python3.7/site-packages (from sagemaker==2.110.0) (4.12.0)\n",
      "Requirement already satisfied: numpy<2.0,>=1.9.0 in /opt/conda/lib/python3.7/site-packages (from sagemaker==2.110.0) (1.21.6)\n",
      "Requirement already satisfied: protobuf<4.0,>=3.1 in /opt/conda/lib/python3.7/site-packages (from sagemaker==2.110.0) (3.20.1)\n",
      "Requirement already satisfied: schema in /opt/conda/lib/python3.7/site-packages (from sagemaker==2.110.0) (0.7.5)\n",
      "Requirement already satisfied: smdebug-rulesconfig==1.0.1 in /opt/conda/lib/python3.7/site-packages (from sagemaker==2.110.0) (1.0.1)\n",
      "Requirement already satisfied: pathos in /opt/conda/lib/python3.7/site-packages (from sagemaker==2.110.0) (0.2.9)\n",
      "Requirement already satisfied: pandas in /opt/conda/lib/python3.7/site-packages (from sagemaker==2.110.0) (1.3.5)\n",
      "Requirement already satisfied: google-pasta in /opt/conda/lib/python3.7/site-packages (from sagemaker==2.110.0) (0.2.0)\n",
      "Requirement already satisfied: protobuf3-to-dict<1.0,>=0.1.5 in /opt/conda/lib/python3.7/site-packages (from sagemaker==2.110.0) (0.1.5)\n",
      "Requirement already satisfied: attrs<22,>=20.3.0 in /opt/conda/lib/python3.7/site-packages (from sagemaker==2.110.0) (21.4.0)\n",
      "Requirement already satisfied: packaging>=20.0 in /opt/conda/lib/python3.7/site-packages (from sagemaker==2.110.0) (20.1)\n",
      "Requirement already satisfied: boto3<2.0,>=1.20.21 in /opt/conda/lib/python3.7/site-packages (from sagemaker==2.110.0) (1.24.62)\n",
      "Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in /opt/conda/lib/python3.7/site-packages (from boto3<2.0,>=1.20.21->sagemaker==2.110.0) (0.6.0)\n",
      "Requirement already satisfied: botocore<1.28.0,>=1.27.62 in /opt/conda/lib/python3.7/site-packages (from boto3<2.0,>=1.20.21->sagemaker==2.110.0) (1.27.62)\n",
      "Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in /opt/conda/lib/python3.7/site-packages (from boto3<2.0,>=1.20.21->sagemaker==2.110.0) (1.0.1)\n",
      "Requirement already satisfied: typing-extensions>=3.6.4 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata<5.0,>=1.4.0->sagemaker==2.110.0) (4.3.0)\n",
      "Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata<5.0,>=1.4.0->sagemaker==2.110.0) (3.8.1)\n",
      "Requirement already satisfied: six in /opt/conda/lib/python3.7/site-packages (from packaging>=20.0->sagemaker==2.110.0) (1.14.0)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging>=20.0->sagemaker==2.110.0) (2.4.6)\n",
      "Requirement already satisfied: python-dateutil>=2.7.3 in /opt/conda/lib/python3.7/site-packages (from pandas->sagemaker==2.110.0) (2.8.1)\n",
      "Requirement already satisfied: pytz>=2017.3 in /opt/conda/lib/python3.7/site-packages (from pandas->sagemaker==2.110.0) (2019.3)\n",
      "Requirement already satisfied: multiprocess>=0.70.13 in /opt/conda/lib/python3.7/site-packages (from pathos->sagemaker==2.110.0) (0.70.13)\n",
      "Requirement already satisfied: pox>=0.3.1 in /opt/conda/lib/python3.7/site-packages (from pathos->sagemaker==2.110.0) (0.3.1)\n",
      "Requirement already satisfied: dill>=0.3.5.1 in /opt/conda/lib/python3.7/site-packages (from pathos->sagemaker==2.110.0) (0.3.5.1)\n",
      "Requirement already satisfied: ppft>=1.7.6.5 in /opt/conda/lib/python3.7/site-packages (from pathos->sagemaker==2.110.0) (1.7.6.5)\n",
      "Requirement already satisfied: contextlib2>=0.5.5 in /opt/conda/lib/python3.7/site-packages (from schema->sagemaker==2.110.0) (0.6.0.post1)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.25.4 in /opt/conda/lib/python3.7/site-packages (from botocore<1.28.0,>=1.27.62->boto3<2.0,>=1.20.21->sagemaker==2.110.0) (1.26.12)\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!pip install --upgrade sagemaker==2.110.0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[33mWARNING: Skipping torchvison as it is not installed.\u001b[0m\u001b[33m\n",
      "\u001b[0m\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0myes: standard output: Broken pipe\n",
      "\u001b[33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\u001b[33m\n",
      "\u001b[0m"
     ]
    }
   ],
   "source": [
    "!yes | pip uninstall torchvison\n",
    "!pip install -qU torchvision\n",
    "!pip install pillow\n",
    "!pip install requests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "# Training using PyTorch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "\n",
    "sagemaker_session = sagemaker.Session()\n",
    "\n",
    "bucket = sagemaker_session.default_bucket()\n",
    "prefix = \"sagemaker/DEMO-pytorch-mnist\"\n",
    "\n",
    "role = sagemaker.get_execution_role()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'carpl-uat-sagemaker-domain'"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import json\n",
    "metadata = json.load(open(\"/opt/ml/metadata/resource-metadata.json\",\"r\"))\n",
    "bucket = metadata[\"UserProfileName\"]\n",
    "bucket"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'sagemaker-ap-south-1-023180687239'"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bucket"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data\n",
    "### Getting the data\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from boto3 import client\n",
    "\n",
    "conn = client('s3')  # again assumes boto.cfg setup, assume AWS S3\n",
    "for key in conn.list_objects(Bucket=bucket)['Contents']:\n",
    "    print(key['Key'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'torchvision'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-a515603fd195>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mtorchvision\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdatasets\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mMNIST\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mtorchvision\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mtransforms\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;31m## Load your data here\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'torchvision'"
     ]
    }
   ],
   "source": [
    "from torchvision.datasets import MNIST\n",
    "from torchvision import transforms\n",
    "\n",
    "## Load your data here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Uploading the data to S3\n",
    "We are going to use the `sagemaker.Session.upload_data` function to upload our datasets to an S3 location. The return value inputs identifies the location -- we will use later when we start the training job.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "input spec (in this case, just an S3 path): s3://sagemaker-ap-south-1-023180687239/sagemaker/DEMO-pytorch-mnist\n"
     ]
    }
   ],
   "source": [
    "inputs = sagemaker_session.upload_data(path=\"data\", bucket=bucket, key_prefix=prefix)\n",
    "print(\"input spec (in this case, just an S3 path): {}\".format(inputs))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train\n",
    "### Training script\n",
    "The `script.py` script provides all the code we need for training and hosting a SageMaker model (`model_fn` function to load a model).\n",
    "The training script is very similar to a training script you might run outside of SageMaker, but you can access useful properties about the training environment through various environment variables, such as:\n",
    "\n",
    "* `SM_MODEL_DIR`: A string representing the path to the directory to write model artifacts to.\n",
    "  These artifacts are uploaded to S3 for model hosting.\n",
    "* `SM_NUM_GPUS`: The number of gpus available in the current container.\n",
    "* `SM_CURRENT_HOST`: The name of the current container on the container network.\n",
    "* `SM_HOSTS`: JSON encoded list containing all the hosts .\n",
    "\n",
    "Supposing one input channel, 'training', was used in the call to the PyTorch estimator's `fit()` method, the following will be set, following the format `SM_CHANNEL_[channel_name]`:\n",
    "\n",
    "* `SM_CHANNEL_TRAINING`: A string representing the path to the directory containing data in the 'training' channel.\n",
    "\n",
    "For more information about training environment variables, please visit [SageMaker Containers](https://github.com/aws/sagemaker-containers).\n",
    "\n",
    "A typical training script loads data from the input channels, configures training with hyperparameters, trains a model, and saves a model to `model_dir` so that it can be hosted later. Hyperparameters are passed to your script as arguments and can be retrieved with an `argparse.ArgumentParser` instance.\n",
    "\n",
    "Because the SageMaker imports the training script, you should put your training code in a main guard (``if __name__=='__main__':``) if you are using the same script to host your model as we do in this example, so that SageMaker does not inadvertently run your training code at the wrong point in execution.\n",
    "\n",
    "For example, the script run by this notebook:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36margparse\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mjson\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mlogging\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mos\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36msys\u001b[39;49;00m\n",
      "\u001b[37m#import sagemaker_containers\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mtorch\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mtorch\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mdistributed\u001b[39;49;00m \u001b[34mas\u001b[39;49;00m \u001b[04m\u001b[36mdist\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mtorch\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mnn\u001b[39;49;00m \u001b[34mas\u001b[39;49;00m \u001b[04m\u001b[36mnn\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mtorch\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mnn\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mfunctional\u001b[39;49;00m \u001b[34mas\u001b[39;49;00m \u001b[04m\u001b[36mF\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mtorch\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36moptim\u001b[39;49;00m \u001b[34mas\u001b[39;49;00m \u001b[04m\u001b[36moptim\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mtorch\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mutils\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mdata\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mtorch\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mutils\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mdata\u001b[39;49;00m\u001b[04m\u001b[36m.\u001b[39;49;00m\u001b[04m\u001b[36mdistributed\u001b[39;49;00m\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mtorchvision\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m datasets, transforms\n",
      "\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mshlex\u001b[39;49;00m, \u001b[04m\u001b[36msubprocess\u001b[39;49;00m\n",
      "\u001b[37m# def install(package):\u001b[39;49;00m\n",
      "\u001b[37m#     os.system(\"pip install \" +  package)\u001b[39;49;00m\n",
      "    \n",
      "\u001b[37m# install('pillow')\u001b[39;49;00m\n",
      "\u001b[37m# install('requests')\u001b[39;49;00m\n",
      "\u001b[37m# install('pydicom')\u001b[39;49;00m\n",
      "\u001b[34mfrom\u001b[39;49;00m \u001b[04m\u001b[36mPIL\u001b[39;49;00m \u001b[34mimport\u001b[39;49;00m Image\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mrequests\u001b[39;49;00m\n",
      "\u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mpydicom\u001b[39;49;00m\n",
      "\n",
      "logger = logging.getLogger(\u001b[31m__name__\u001b[39;49;00m)\n",
      "logger.setLevel(logging.DEBUG)\n",
      "logger.addHandler(logging.StreamHandler(sys.stdout))\n",
      "\n",
      "\n",
      "\u001b[37m# Based on https://github.com/pytorch/examples/blob/master/mnist/main.py\u001b[39;49;00m\n",
      "\u001b[34mclass\u001b[39;49;00m \u001b[04m\u001b[32mNet\u001b[39;49;00m(nn.Module):\n",
      "    \u001b[34mdef\u001b[39;49;00m \u001b[32m__init__\u001b[39;49;00m(\u001b[36mself\u001b[39;49;00m):\n",
      "        \u001b[36msuper\u001b[39;49;00m(Net, \u001b[36mself\u001b[39;49;00m).\u001b[32m__init__\u001b[39;49;00m()\n",
      "        \u001b[36mself\u001b[39;49;00m.conv1 = nn.Conv2d(\u001b[34m1\u001b[39;49;00m, \u001b[34m10\u001b[39;49;00m, kernel_size=\u001b[34m5\u001b[39;49;00m)\n",
      "        \u001b[36mself\u001b[39;49;00m.conv2 = nn.Conv2d(\u001b[34m10\u001b[39;49;00m, \u001b[34m20\u001b[39;49;00m, kernel_size=\u001b[34m5\u001b[39;49;00m)\n",
      "        \u001b[36mself\u001b[39;49;00m.conv2_drop = nn.Dropout2d()\n",
      "        \u001b[36mself\u001b[39;49;00m.fc1 = nn.Linear(\u001b[34m320\u001b[39;49;00m, \u001b[34m50\u001b[39;49;00m)\n",
      "        \u001b[36mself\u001b[39;49;00m.fc2 = nn.Linear(\u001b[34m50\u001b[39;49;00m, \u001b[34m10\u001b[39;49;00m)\n",
      "\n",
      "    \u001b[34mdef\u001b[39;49;00m \u001b[32mforward\u001b[39;49;00m(\u001b[36mself\u001b[39;49;00m, x):\n",
      "        x = F.relu(F.max_pool2d(\u001b[36mself\u001b[39;49;00m.conv1(x), \u001b[34m2\u001b[39;49;00m))\n",
      "        x = F.relu(F.max_pool2d(\u001b[36mself\u001b[39;49;00m.conv2_drop(\u001b[36mself\u001b[39;49;00m.conv2(x)), \u001b[34m2\u001b[39;49;00m))\n",
      "        x = x.view(-\u001b[34m1\u001b[39;49;00m, \u001b[34m320\u001b[39;49;00m)\n",
      "        x = F.relu(\u001b[36mself\u001b[39;49;00m.fc1(x))\n",
      "        x = F.dropout(x, training=\u001b[36mself\u001b[39;49;00m.training)\n",
      "        x = \u001b[36mself\u001b[39;49;00m.fc2(x)\n",
      "        \u001b[34mreturn\u001b[39;49;00m F.log_softmax(x, dim=\u001b[34m1\u001b[39;49;00m)\n",
      "\n",
      "\n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32m_get_train_data_loader\u001b[39;49;00m(batch_size, training_dir, is_distributed, **kwargs):\n",
      "    logger.info(\u001b[33m\"\u001b[39;49;00m\u001b[33mGet train data loader\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\n",
      "    dataset = datasets.MNIST(\n",
      "        training_dir,\n",
      "        train=\u001b[34mTrue\u001b[39;49;00m,\n",
      "        transform=transforms.Compose(\n",
      "            [transforms.ToTensor(), transforms.Normalize((\u001b[34m0.1307\u001b[39;49;00m,), (\u001b[34m0.3081\u001b[39;49;00m,))]\n",
      "        ),\n",
      "    )\n",
      "    train_sampler = (\n",
      "        torch.utils.data.distributed.DistributedSampler(dataset) \u001b[34mif\u001b[39;49;00m is_distributed \u001b[34melse\u001b[39;49;00m \u001b[34mNone\u001b[39;49;00m\n",
      "    )\n",
      "    \u001b[34mreturn\u001b[39;49;00m torch.utils.data.DataLoader(\n",
      "        dataset,\n",
      "        batch_size=batch_size,\n",
      "        shuffle=train_sampler \u001b[35mis\u001b[39;49;00m \u001b[34mNone\u001b[39;49;00m,\n",
      "        sampler=train_sampler,\n",
      "        **kwargs\n",
      "    )\n",
      "\n",
      "\n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32m_get_test_data_loader\u001b[39;49;00m(test_batch_size, training_dir, **kwargs):\n",
      "    logger.info(\u001b[33m\"\u001b[39;49;00m\u001b[33mGet test data loader\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\n",
      "    \u001b[34mreturn\u001b[39;49;00m torch.utils.data.DataLoader(\n",
      "        datasets.MNIST(\n",
      "            training_dir,\n",
      "            train=\u001b[34mFalse\u001b[39;49;00m,\n",
      "            transform=transforms.Compose(\n",
      "                [transforms.ToTensor(), transforms.Normalize((\u001b[34m0.1307\u001b[39;49;00m,), (\u001b[34m0.3081\u001b[39;49;00m,))]\n",
      "            ),\n",
      "        ),\n",
      "        batch_size=test_batch_size,\n",
      "        shuffle=\u001b[34mTrue\u001b[39;49;00m,\n",
      "        **kwargs\n",
      "    )\n",
      "\n",
      "\n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32m_average_gradients\u001b[39;49;00m(model):\n",
      "    \u001b[37m# Gradient averaging.\u001b[39;49;00m\n",
      "    size = \u001b[36mfloat\u001b[39;49;00m(dist.get_world_size())\n",
      "    \u001b[34mfor\u001b[39;49;00m param \u001b[35min\u001b[39;49;00m model.parameters():\n",
      "        dist.all_reduce(param.grad.data, op=dist.reduce_op.SUM)\n",
      "        param.grad.data /= size\n",
      "\n",
      "\n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32mtrain\u001b[39;49;00m(args):\n",
      "    is_distributed = \u001b[36mlen\u001b[39;49;00m(args.hosts) > \u001b[34m1\u001b[39;49;00m \u001b[35mand\u001b[39;49;00m args.backend \u001b[35mis\u001b[39;49;00m \u001b[35mnot\u001b[39;49;00m \u001b[34mNone\u001b[39;49;00m\n",
      "    logger.debug(\u001b[33m\"\u001b[39;49;00m\u001b[33mDistributed training - \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m.format(is_distributed))\n",
      "    use_cuda = args.num_gpus > \u001b[34m0\u001b[39;49;00m\n",
      "    logger.debug(\u001b[33m\"\u001b[39;49;00m\u001b[33mNumber of gpus available - \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m.format(args.num_gpus))\n",
      "    kwargs = {\u001b[33m\"\u001b[39;49;00m\u001b[33mnum_workers\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: \u001b[34m1\u001b[39;49;00m, \u001b[33m\"\u001b[39;49;00m\u001b[33mpin_memory\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m: \u001b[34mTrue\u001b[39;49;00m} \u001b[34mif\u001b[39;49;00m use_cuda \u001b[34melse\u001b[39;49;00m {}\n",
      "    device = torch.device(\u001b[33m\"\u001b[39;49;00m\u001b[33mcuda\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m \u001b[34mif\u001b[39;49;00m use_cuda \u001b[34melse\u001b[39;49;00m \u001b[33m\"\u001b[39;49;00m\u001b[33mcpu\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\n",
      "\n",
      "    \u001b[34mif\u001b[39;49;00m is_distributed:\n",
      "        \u001b[37m# Initialize the distributed environment.\u001b[39;49;00m\n",
      "        world_size = \u001b[36mlen\u001b[39;49;00m(args.hosts)\n",
      "        os.environ[\u001b[33m\"\u001b[39;49;00m\u001b[33mWORLD_SIZE\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m] = \u001b[36mstr\u001b[39;49;00m(world_size)\n",
      "        host_rank = args.hosts.index(args.current_host)\n",
      "        os.environ[\u001b[33m\"\u001b[39;49;00m\u001b[33mRANK\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m] = \u001b[36mstr\u001b[39;49;00m(host_rank)\n",
      "        dist.init_process_group(backend=args.backend, rank=host_rank, world_size=world_size)\n",
      "        logger.info(\n",
      "            \u001b[33m\"\u001b[39;49;00m\u001b[33mInitialized the distributed environment: \u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\u001b[33m backend on \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m nodes. \u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m.format(\n",
      "                args.backend, dist.get_world_size()\n",
      "            )\n",
      "            + \u001b[33m\"\u001b[39;49;00m\u001b[33mCurrent host rank is \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m. Number of gpus: \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m.format(dist.get_rank(), args.num_gpus)\n",
      "        )\n",
      "\n",
      "    \u001b[37m# set the seed for generating random numbers\u001b[39;49;00m\n",
      "    torch.manual_seed(args.seed)\n",
      "    \u001b[34mif\u001b[39;49;00m use_cuda:\n",
      "        torch.cuda.manual_seed(args.seed)\n",
      "\n",
      "    train_loader = _get_train_data_loader(args.batch_size, args.data_dir, is_distributed, **kwargs)\n",
      "    test_loader = _get_test_data_loader(args.test_batch_size, args.data_dir, **kwargs)\n",
      "\n",
      "    logger.debug(\n",
      "        \u001b[33m\"\u001b[39;49;00m\u001b[33mProcesses \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m/\u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m (\u001b[39;49;00m\u001b[33m{:.0f}\u001b[39;49;00m\u001b[33m%\u001b[39;49;00m\u001b[33m) of train data\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m.format(\n",
      "            \u001b[36mlen\u001b[39;49;00m(train_loader.sampler),\n",
      "            \u001b[36mlen\u001b[39;49;00m(train_loader.dataset),\n",
      "            \u001b[34m100.0\u001b[39;49;00m * \u001b[36mlen\u001b[39;49;00m(train_loader.sampler) / \u001b[36mlen\u001b[39;49;00m(train_loader.dataset),\n",
      "        )\n",
      "    )\n",
      "\n",
      "    logger.debug(\n",
      "        \u001b[33m\"\u001b[39;49;00m\u001b[33mProcesses \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m/\u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m (\u001b[39;49;00m\u001b[33m{:.0f}\u001b[39;49;00m\u001b[33m%\u001b[39;49;00m\u001b[33m) of test data\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m.format(\n",
      "            \u001b[36mlen\u001b[39;49;00m(test_loader.sampler),\n",
      "            \u001b[36mlen\u001b[39;49;00m(test_loader.dataset),\n",
      "            \u001b[34m100.0\u001b[39;49;00m * \u001b[36mlen\u001b[39;49;00m(test_loader.sampler) / \u001b[36mlen\u001b[39;49;00m(test_loader.dataset),\n",
      "        )\n",
      "    )\n",
      "\n",
      "    model = Net().to(device)\n",
      "    \u001b[34mif\u001b[39;49;00m is_distributed \u001b[35mand\u001b[39;49;00m use_cuda:\n",
      "        \u001b[37m# multi-machine multi-gpu case\u001b[39;49;00m\n",
      "        model = torch.nn.parallel.DistributedDataParallel(model)\n",
      "    \u001b[34melse\u001b[39;49;00m:\n",
      "        \u001b[37m# single-machine multi-gpu case or single-machine or multi-machine cpu case\u001b[39;49;00m\n",
      "        model = torch.nn.DataParallel(model)\n",
      "\n",
      "    optimizer = optim.SGD(model.parameters(), lr=args.lr, momentum=args.momentum)\n",
      "\n",
      "    \u001b[34mfor\u001b[39;49;00m epoch \u001b[35min\u001b[39;49;00m \u001b[36mrange\u001b[39;49;00m(\u001b[34m1\u001b[39;49;00m, args.epochs + \u001b[34m1\u001b[39;49;00m):\n",
      "        model.train()\n",
      "        \u001b[34mfor\u001b[39;49;00m batch_idx, (data, target) \u001b[35min\u001b[39;49;00m \u001b[36menumerate\u001b[39;49;00m(train_loader, \u001b[34m1\u001b[39;49;00m):\n",
      "            data, target = data.to(device), target.to(device)\n",
      "            optimizer.zero_grad()\n",
      "            output = model(data)\n",
      "            loss = F.nll_loss(output, target)\n",
      "            loss.backward()\n",
      "            \u001b[34mif\u001b[39;49;00m is_distributed \u001b[35mand\u001b[39;49;00m \u001b[35mnot\u001b[39;49;00m use_cuda:\n",
      "                \u001b[37m# average gradients manually for multi-machine cpu case only\u001b[39;49;00m\n",
      "                _average_gradients(model)\n",
      "            optimizer.step()\n",
      "            \u001b[34mif\u001b[39;49;00m batch_idx % args.log_interval == \u001b[34m0\u001b[39;49;00m:\n",
      "                logger.info(\n",
      "                    \u001b[33m\"\u001b[39;49;00m\u001b[33mTrain Epoch: \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m [\u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m/\u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m (\u001b[39;49;00m\u001b[33m{:.0f}\u001b[39;49;00m\u001b[33m%\u001b[39;49;00m\u001b[33m)] Loss: \u001b[39;49;00m\u001b[33m{:.6f}\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m.format(\n",
      "                        epoch,\n",
      "                        batch_idx * \u001b[36mlen\u001b[39;49;00m(data),\n",
      "                        \u001b[36mlen\u001b[39;49;00m(train_loader.sampler),\n",
      "                        \u001b[34m100.0\u001b[39;49;00m * batch_idx / \u001b[36mlen\u001b[39;49;00m(train_loader),\n",
      "                        loss.item(),\n",
      "                    )\n",
      "                )\n",
      "        test(model, test_loader, device)\n",
      "    save_model(model, args.model_dir)\n",
      "\n",
      "\n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32mtest\u001b[39;49;00m(model, test_loader, device):\n",
      "    model.eval()\n",
      "    test_loss = \u001b[34m0\u001b[39;49;00m\n",
      "    correct = \u001b[34m0\u001b[39;49;00m\n",
      "    \u001b[34mwith\u001b[39;49;00m torch.no_grad():\n",
      "        \u001b[34mfor\u001b[39;49;00m data, target \u001b[35min\u001b[39;49;00m test_loader:\n",
      "            data, target = data.to(device), target.to(device)\n",
      "            output = model(data)\n",
      "            test_loss += F.nll_loss(output, target, size_average=\u001b[34mFalse\u001b[39;49;00m).item()  \u001b[37m# sum up batch loss\u001b[39;49;00m\n",
      "            pred = output.max(\u001b[34m1\u001b[39;49;00m, keepdim=\u001b[34mTrue\u001b[39;49;00m)[\u001b[34m1\u001b[39;49;00m]  \u001b[37m# get the index of the max log-probability\u001b[39;49;00m\n",
      "            correct += pred.eq(target.view_as(pred)).sum().item()\n",
      "\n",
      "    test_loss /= \u001b[36mlen\u001b[39;49;00m(test_loader.dataset)\n",
      "    logger.info(\n",
      "        \u001b[33m\"\u001b[39;49;00m\u001b[33mTest set: Average loss: \u001b[39;49;00m\u001b[33m{:.4f}\u001b[39;49;00m\u001b[33m, Accuracy: \u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m/\u001b[39;49;00m\u001b[33m{}\u001b[39;49;00m\u001b[33m (\u001b[39;49;00m\u001b[33m{:.0f}\u001b[39;49;00m\u001b[33m%\u001b[39;49;00m\u001b[33m)\u001b[39;49;00m\u001b[33m\\n\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m.format(\n",
      "            test_loss, correct, \u001b[36mlen\u001b[39;49;00m(test_loader.dataset), \u001b[34m100.0\u001b[39;49;00m * correct / \u001b[36mlen\u001b[39;49;00m(test_loader.dataset)\n",
      "        )\n",
      "    )\n",
      "\n",
      "\n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32mmodel_fn\u001b[39;49;00m(model_dir):\n",
      "    device = torch.device(\u001b[33m\"\u001b[39;49;00m\u001b[33mcuda\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m \u001b[34mif\u001b[39;49;00m torch.cuda.is_available() \u001b[34melse\u001b[39;49;00m \u001b[33m\"\u001b[39;49;00m\u001b[33mcpu\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\n",
      "    model = torch.nn.DataParallel(Net())\n",
      "    \u001b[34mwith\u001b[39;49;00m \u001b[36mopen\u001b[39;49;00m(os.path.join(model_dir, \u001b[33m\"\u001b[39;49;00m\u001b[33mmodel.pth\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m), \u001b[33m\"\u001b[39;49;00m\u001b[33mrb\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m) \u001b[34mas\u001b[39;49;00m f:\n",
      "        model.load_state_dict(torch.load(f))\n",
      "    \u001b[34mreturn\u001b[39;49;00m model.to(device)\n",
      "\n",
      "\u001b[37m# def input_fn(request_body, request_content_type):\u001b[39;49;00m\n",
      "\u001b[37m#     device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\u001b[39;49;00m\n",
      "\u001b[37m#     assert request_content_type=='application/x-npy'\u001b[39;49;00m\n",
      "\u001b[37m#     # data = json.loads(request_body)['inputs']\u001b[39;49;00m\n",
      "\u001b[37m#     data = request_body\u001b[39;49;00m\n",
      "\u001b[37m#     data = torch.tensor(data, dtype=torch.float32, device=device)\u001b[39;49;00m\n",
      "\u001b[37m#     return data\u001b[39;49;00m\n",
      "\n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32minput_fn\u001b[39;49;00m(request_body, content_type=\u001b[33m'\u001b[39;49;00m\u001b[33mapplication/json\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m):\n",
      "    logger.info(\u001b[33m'\u001b[39;49;00m\u001b[33mDeserializing the input data.\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\n",
      "    \u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mrequests\u001b[39;49;00m\n",
      "    \u001b[34mimport\u001b[39;49;00m \u001b[04m\u001b[36mnumpy\u001b[39;49;00m \u001b[34mas\u001b[39;49;00m \u001b[04m\u001b[36mnp\u001b[39;49;00m\n",
      "    device = torch.device(\u001b[33m\"\u001b[39;49;00m\u001b[33mcuda\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m \u001b[34mif\u001b[39;49;00m torch.cuda.is_available() \u001b[34melse\u001b[39;49;00m \u001b[33m\"\u001b[39;49;00m\u001b[33mcpu\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\n",
      "    \u001b[34mif\u001b[39;49;00m content_type == \u001b[33m'\u001b[39;49;00m\u001b[33mapplication/json\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m:\n",
      "        \u001b[37m# request_body = json.dumps({\"url\":\"http://rasbt.github.io/mlxtend/user_guide/data/mnist_data_files/mnist_data_10_0.png\"})    \u001b[39;49;00m\n",
      "        input_data = json.loads(request_body)\n",
      "        url = input_data[\u001b[33m'\u001b[39;49;00m\u001b[33murl\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m]\n",
      "        logger.info(\u001b[33mf\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\u001b[33mImage url: \u001b[39;49;00m\u001b[33m{\u001b[39;49;00murl\u001b[33m}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\n",
      "        device = torch.device(\u001b[33m\"\u001b[39;49;00m\u001b[33mcuda\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m \u001b[34mif\u001b[39;49;00m torch.cuda.is_available() \u001b[34melse\u001b[39;49;00m \u001b[33m\"\u001b[39;49;00m\u001b[33mcpu\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\n",
      "        \u001b[37m# resp = requests.get(url, stream=True)\u001b[39;49;00m\n",
      "        \u001b[37m# raw_data = resp.raw\u001b[39;49;00m\n",
      "        image_data = Image.open(requests.get(url, stream=\u001b[34mTrue\u001b[39;49;00m).raw).convert(\u001b[33m'\u001b[39;49;00m\u001b[33mL\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\n",
      "\n",
      "        image_transform = transforms.Compose([\n",
      "            transforms.ToTensor(),\n",
      "            transforms.Resize((\u001b[34m28\u001b[39;49;00m,\u001b[34m28\u001b[39;49;00m)),\n",
      "        ])\n",
      "        \u001b[36mprint\u001b[39;49;00m(image_data)\n",
      "        data = image_transform(image_data)\n",
      "        \u001b[36mprint\u001b[39;49;00m(data.shape)\n",
      "        data = np.asarray(data)\n",
      "        \u001b[36mprint\u001b[39;49;00m(data.shape)\n",
      "        data = torch.tensor(data, dtype=torch.float32, device=device)\n",
      "        \u001b[36mprint\u001b[39;49;00m(data.shape)\n",
      "        \u001b[34mreturn\u001b[39;49;00m data\n",
      "    \u001b[34mraise\u001b[39;49;00m \u001b[36mException\u001b[39;49;00m(\u001b[33mf\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\u001b[33mRequested unsupported ContentType in content_type \u001b[39;49;00m\u001b[33m{\u001b[39;49;00mcontent_type\u001b[33m}\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m)\n",
      "    \n",
      "    \n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32msave_model\u001b[39;49;00m(model, model_dir):\n",
      "    logger.info(\u001b[33m\"\u001b[39;49;00m\u001b[33mSaving the model.\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\n",
      "    path = os.path.join(model_dir, \u001b[33m\"\u001b[39;49;00m\u001b[33mmodel.pth\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\n",
      "    \u001b[37m# recommended way from http://pytorch.org/docs/master/notes/serialization.html\u001b[39;49;00m\n",
      "    torch.save(model.cpu().state_dict(), path)\n",
      "\n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32mpredict_fn\u001b[39;49;00m(input_object, model):\n",
      "    \u001b[34mwith\u001b[39;49;00m torch.no_grad():\n",
      "        prediction = model(input_object)\n",
      "\n",
      "    \u001b[34mreturn\u001b[39;49;00m prediction\n",
      "\n",
      "\u001b[34mdef\u001b[39;49;00m \u001b[32moutput_fn\u001b[39;49;00m(predictions, content_type):\n",
      "    \u001b[34massert\u001b[39;49;00m content_type == \u001b[33m'\u001b[39;49;00m\u001b[33mapplication/json\u001b[39;49;00m\u001b[33m'\u001b[39;49;00m\n",
      "    res = predictions.cpu().numpy().tolist()\n",
      "    \u001b[34mreturn\u001b[39;49;00m json.dumps(res)\n",
      "\n",
      "\u001b[34mif\u001b[39;49;00m \u001b[31m__name__\u001b[39;49;00m == \u001b[33m\"\u001b[39;49;00m\u001b[33m__main__\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m:\n",
      "    parser = argparse.ArgumentParser()\n",
      "\n",
      "    \u001b[37m# Data and model checkpoints directories\u001b[39;49;00m\n",
      "    parser.add_argument(\n",
      "        \u001b[33m\"\u001b[39;49;00m\u001b[33m--batch-size\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\n",
      "        \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m,\n",
      "        default=\u001b[34m64\u001b[39;49;00m,\n",
      "        metavar=\u001b[33m\"\u001b[39;49;00m\u001b[33mN\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\n",
      "        help=\u001b[33m\"\u001b[39;49;00m\u001b[33minput batch size for training (default: 64)\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\n",
      "    )\n",
      "    parser.add_argument(\n",
      "        \u001b[33m\"\u001b[39;49;00m\u001b[33m--test-batch-size\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\n",
      "        \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m,\n",
      "        default=\u001b[34m1000\u001b[39;49;00m,\n",
      "        metavar=\u001b[33m\"\u001b[39;49;00m\u001b[33mN\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\n",
      "        help=\u001b[33m\"\u001b[39;49;00m\u001b[33minput batch size for testing (default: 1000)\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\n",
      "    )\n",
      "    parser.add_argument(\n",
      "        \u001b[33m\"\u001b[39;49;00m\u001b[33m--epochs\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\n",
      "        \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m,\n",
      "        default=\u001b[34m10\u001b[39;49;00m,\n",
      "        metavar=\u001b[33m\"\u001b[39;49;00m\u001b[33mN\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\n",
      "        help=\u001b[33m\"\u001b[39;49;00m\u001b[33mnumber of epochs to train (default: 10)\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\n",
      "    )\n",
      "    parser.add_argument(\n",
      "        \u001b[33m\"\u001b[39;49;00m\u001b[33m--lr\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mfloat\u001b[39;49;00m, default=\u001b[34m0.01\u001b[39;49;00m, metavar=\u001b[33m\"\u001b[39;49;00m\u001b[33mLR\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, help=\u001b[33m\"\u001b[39;49;00m\u001b[33mlearning rate (default: 0.01)\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\n",
      "    )\n",
      "    parser.add_argument(\n",
      "        \u001b[33m\"\u001b[39;49;00m\u001b[33m--momentum\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mfloat\u001b[39;49;00m, default=\u001b[34m0.5\u001b[39;49;00m, metavar=\u001b[33m\"\u001b[39;49;00m\u001b[33mM\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, help=\u001b[33m\"\u001b[39;49;00m\u001b[33mSGD momentum (default: 0.5)\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m\n",
      "    )\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--seed\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m, default=\u001b[34m1\u001b[39;49;00m, metavar=\u001b[33m\"\u001b[39;49;00m\u001b[33mS\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, help=\u001b[33m\"\u001b[39;49;00m\u001b[33mrandom seed (default: 1)\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m)\n",
      "    parser.add_argument(\n",
      "        \u001b[33m\"\u001b[39;49;00m\u001b[33m--log-interval\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\n",
      "        \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m,\n",
      "        default=\u001b[34m100\u001b[39;49;00m,\n",
      "        metavar=\u001b[33m\"\u001b[39;49;00m\u001b[33mN\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\n",
      "        help=\u001b[33m\"\u001b[39;49;00m\u001b[33mhow many batches to wait before logging training status\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\n",
      "    )\n",
      "    parser.add_argument(\n",
      "        \u001b[33m\"\u001b[39;49;00m\u001b[33m--backend\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\n",
      "        \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m,\n",
      "        default=\u001b[34mNone\u001b[39;49;00m,\n",
      "        help=\u001b[33m\"\u001b[39;49;00m\u001b[33mbackend for distributed training (tcp, gloo on cpu and gloo, nccl on gpu)\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m,\n",
      "    )\n",
      "\n",
      "    \u001b[37m# Container environment\u001b[39;49;00m\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--hosts\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mlist\u001b[39;49;00m, default=json.loads(os.environ[\u001b[33m\"\u001b[39;49;00m\u001b[33mSM_HOSTS\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m]))\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--current-host\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, default=os.environ[\u001b[33m\"\u001b[39;49;00m\u001b[33mSM_CURRENT_HOST\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m])\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--model-dir\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, default=os.environ[\u001b[33m\"\u001b[39;49;00m\u001b[33mSM_MODEL_DIR\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m])\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--data-dir\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mstr\u001b[39;49;00m, default=os.environ[\u001b[33m\"\u001b[39;49;00m\u001b[33mSM_CHANNEL_TRAINING\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m])\n",
      "    parser.add_argument(\u001b[33m\"\u001b[39;49;00m\u001b[33m--num-gpus\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m, \u001b[36mtype\u001b[39;49;00m=\u001b[36mint\u001b[39;49;00m, default=os.environ[\u001b[33m\"\u001b[39;49;00m\u001b[33mSM_NUM_GPUS\u001b[39;49;00m\u001b[33m\"\u001b[39;49;00m])\n",
      "\n",
      "    \n",
      "\n",
      "    train(parser.parse_args())\n"
     ]
    }
   ],
   "source": [
    "!pygmentize code/script.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Run training in SageMaker\n",
    "\n",
    "The `PyTorch` class allows us to run our training function as a training job on SageMaker infrastructure. We need to configure it with our training script, an IAM role, the number of training instances, the training instance type, and hyperparameters. In this case we are going to run our training job on 2 ```ml.c4.xlarge``` instances. But this example can be ran on one or multiple, cpu or gpu instances ([full list of available instances](https://aws.amazon.com/sagemaker/pricing/instance-types/)). The hyperparameters parameter is a dict of values that will be passed to your training script -- you can see how to access these values in the `mnist.py` script above.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.pytorch import PyTorch"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Version: 2.110.0\n"
     ]
    }
   ],
   "source": [
    "!pip show sagemaker | grep Version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.pytorch import PyTorch\n",
    "\n",
    "estimator = PyTorch(\n",
    "    entry_point=\"mnist.py\",\n",
    "    role=role,\n",
    "    py_version=\"py38\",\n",
    "    framework_version=\"1.11.0\",\n",
    "    instance_count=2,\n",
    "    instance_type=\"ml.c5.2xlarge\",\n",
    "    hyperparameters={\"epochs\": 1, \"backend\": \"gloo\"},\n",
    "    dependencies=['code/requirements.txt'],\n",
    "    source_dir = \"code\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After we've constructed our `PyTorch` object, we can fit it using the data we uploaded to S3. SageMaker makes sure our data is available in the local filesystem, so our training script can simply read the data from disk.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2022-10-11 07:14:12 Starting - Starting the training job...\n",
      "2022-10-11 07:14:36 Starting - Preparing the instances for trainingProfilerReport-1665472452: InProgress\n",
      "......\n",
      "2022-10-11 07:15:36 Downloading - Downloading input data...\n",
      "2022-10-11 07:15:56 Training - Downloading the training image...\n",
      "2022-10-11 07:16:43 Training - Training image download completed. Training in progress..\u001b[35mbash: cannot set terminal process group (-1): Inappropriate ioctl for device\u001b[0m\n",
      "\u001b[35mbash: no job control in this shell\u001b[0m\n",
      "\u001b[35m2022-10-11 07:16:44,908 sagemaker-training-toolkit INFO     Imported framework sagemaker_pytorch_container.training\u001b[0m\n",
      "\u001b[35m2022-10-11 07:16:44,910 sagemaker-training-toolkit INFO     No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[35m2022-10-11 07:16:44,916 sagemaker_pytorch_container.training INFO     Block until all host DNS lookups succeed.\u001b[0m\n",
      "\u001b[35m2022-10-11 07:16:44,922 sagemaker_pytorch_container.training INFO     Invoking user training script.\u001b[0m\n",
      "\u001b[35m2022-10-11 07:16:45,353 sagemaker-training-toolkit INFO     Installing dependencies from requirements.txt:\u001b[0m\n",
      "\u001b[35m/opt/conda/bin/python3.8 -m pip install -r requirements.txt\u001b[0m\n",
      "\u001b[35mRequirement already satisfied: requests in /opt/conda/lib/python3.8/site-packages (from -r requirements.txt (line 1)) (2.28.1)\u001b[0m\n",
      "\u001b[35mRequirement already satisfied: pillow in /opt/conda/lib/python3.8/site-packages (from -r requirements.txt (line 2)) (9.2.0)\u001b[0m\n",
      "\u001b[35mCollecting pydicom\u001b[0m\n",
      "\u001b[35mDownloading pydicom-2.3.0-py3-none-any.whl (2.0 MB)\u001b[0m\n",
      "\u001b[35m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 79.8 MB/s eta 0:00:00\u001b[0m\n",
      "\u001b[35mRequirement already satisfied: numpy in /opt/conda/lib/python3.8/site-packages (from -r requirements.txt (line 4)) (1.22.2)\u001b[0m\n",
      "\u001b[35mRequirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.8/site-packages (from requests->-r requirements.txt (line 1)) (3.4)\u001b[0m\n",
      "\u001b[35mRequirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.8/site-packages (from requests->-r requirements.txt (line 1)) (1.26.11)\u001b[0m\n",
      "\u001b[35mRequirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.8/site-packages (from requests->-r requirements.txt (line 1)) (2022.9.24)\u001b[0m\n",
      "\u001b[35mRequirement already satisfied: charset-normalizer<3,>=2 in /opt/conda/lib/python3.8/site-packages (from requests->-r requirements.txt (line 1)) (2.1.1)\u001b[0m\n",
      "\u001b[34mbash: cannot set terminal process group (-1): Inappropriate ioctl for device\u001b[0m\n",
      "\u001b[34mbash: no job control in this shell\u001b[0m\n",
      "\u001b[34m2022-10-11 07:16:44,807 sagemaker-training-toolkit INFO     Imported framework sagemaker_pytorch_container.training\u001b[0m\n",
      "\u001b[34m2022-10-11 07:16:44,809 sagemaker-training-toolkit INFO     No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[34m2022-10-11 07:16:44,815 sagemaker_pytorch_container.training INFO     Block until all host DNS lookups succeed.\u001b[0m\n",
      "\u001b[34m2022-10-11 07:16:44,821 sagemaker_pytorch_container.training INFO     Invoking user training script.\u001b[0m\n",
      "\u001b[34m2022-10-11 07:16:45,266 sagemaker-training-toolkit INFO     Installing dependencies from requirements.txt:\u001b[0m\n",
      "\u001b[34m/opt/conda/bin/python3.8 -m pip install -r requirements.txt\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: requests in /opt/conda/lib/python3.8/site-packages (from -r requirements.txt (line 1)) (2.28.1)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: pillow in /opt/conda/lib/python3.8/site-packages (from -r requirements.txt (line 2)) (9.2.0)\u001b[0m\n",
      "\u001b[34mCollecting pydicom\u001b[0m\n",
      "\u001b[34mDownloading pydicom-2.3.0-py3-none-any.whl (2.0 MB)\u001b[0m\n",
      "\u001b[34m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.0/2.0 MB 71.8 MB/s eta 0:00:00\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: numpy in /opt/conda/lib/python3.8/site-packages (from -r requirements.txt (line 4)) (1.22.2)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: charset-normalizer<3,>=2 in /opt/conda/lib/python3.8/site-packages (from requests->-r requirements.txt (line 1)) (2.1.1)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: urllib3<1.27,>=1.21.1 in /opt/conda/lib/python3.8/site-packages (from requests->-r requirements.txt (line 1)) (1.26.11)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.8/site-packages (from requests->-r requirements.txt (line 1)) (3.4)\u001b[0m\n",
      "\u001b[34mRequirement already satisfied: certifi>=2017.4.17 in /opt/conda/lib/python3.8/site-packages (from requests->-r requirements.txt (line 1)) (2022.9.24)\u001b[0m\n",
      "\u001b[34mInstalling collected packages: pydicom\u001b[0m\n",
      "\u001b[35mInstalling collected packages: pydicom\u001b[0m\n",
      "\u001b[35mSuccessfully installed pydicom-2.3.0\u001b[0m\n",
      "\u001b[35mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\n",
      "\u001b[34mSuccessfully installed pydicom-2.3.0\u001b[0m\n",
      "\u001b[34mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\u001b[0m\n",
      "\u001b[34m2022-10-11 07:16:47,914 sagemaker-training-toolkit INFO     Waiting for the process to finish and give a return code.\u001b[0m\n",
      "\u001b[34m2022-10-11 07:16:47,914 sagemaker-training-toolkit INFO     Done waiting for a return code. Received 0 from exiting process.\u001b[0m\n",
      "\u001b[34m2022-10-11 07:16:47,917 sagemaker-training-toolkit INFO     No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[34m2022-10-11 07:16:47,925 sagemaker-training-toolkit INFO     No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[34m2022-10-11 07:16:47,935 sagemaker-training-toolkit INFO     No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[34m2022-10-11 07:16:47,941 sagemaker-training-toolkit INFO     Invoking user script\u001b[0m\n",
      "\u001b[34mTraining Env:\u001b[0m\n",
      "\u001b[34m{\n",
      "    \"additional_framework_parameters\": {},\n",
      "    \"channel_input_dirs\": {\n",
      "        \"training\": \"/opt/ml/input/data/training\"\n",
      "    },\n",
      "    \"current_host\": \"algo-1\",\n",
      "    \"current_instance_group\": \"homogeneousCluster\",\n",
      "    \"current_instance_group_hosts\": [\n",
      "        \"algo-2\",\n",
      "        \"algo-1\"\n",
      "    ],\n",
      "    \"current_instance_type\": \"ml.c5.2xlarge\",\n",
      "    \"distribution_hosts\": [],\n",
      "    \"distribution_instance_groups\": [],\n",
      "    \"framework_module\": \"sagemaker_pytorch_container.training:main\",\n",
      "    \"hosts\": [\n",
      "        \"algo-1\",\n",
      "        \"algo-2\"\n",
      "    ],\n",
      "    \"hyperparameters\": {\n",
      "        \"backend\": \"gloo\",\n",
      "        \"epochs\": 1\n",
      "    },\n",
      "    \"input_config_dir\": \"/opt/ml/input/config\",\n",
      "    \"input_data_config\": {\n",
      "        \"training\": {\n",
      "            \"TrainingInputMode\": \"File\",\n",
      "            \"S3DistributionType\": \"FullyReplicated\",\n",
      "            \"RecordWrapperType\": \"None\"\n",
      "        }\n",
      "    },\n",
      "    \"input_dir\": \"/opt/ml/input\",\n",
      "    \"instance_groups\": [\n",
      "        \"homogeneousCluster\"\n",
      "    ],\n",
      "    \"instance_groups_dict\": {\n",
      "        \"homogeneousCluster\": {\n",
      "            \"instance_group_name\": \"homogeneousCluster\",\n",
      "            \"instance_type\": \"ml.c5.2xlarge\",\n",
      "            \"hosts\": [\n",
      "                \"algo-2\",\n",
      "                \"algo-1\"\n",
      "            ]\n",
      "        }\n",
      "    },\n",
      "    \"is_hetero\": false,\n",
      "    \"is_master\": true,\n",
      "    \"is_modelparallel_enabled\": null,\n",
      "    \"job_name\": \"pytorch-training-2022-10-11-07-14-12-085\",\n",
      "    \"log_level\": 20,\n",
      "    \"master_hostname\": \"algo-1\",\n",
      "    \"model_dir\": \"/opt/ml/model\",\n",
      "    \"module_dir\": \"s3://sagemaker-ap-south-1-023180687239/pytorch-training-2022-10-11-07-14-12-085/source/sourcedir.tar.gz\",\n",
      "    \"module_name\": \"mnist\",\n",
      "    \"network_interface_name\": \"eth0\",\n",
      "    \"num_cpus\": 8,\n",
      "    \"num_gpus\": 0,\n",
      "    \"output_data_dir\": \"/opt/ml/output/data\",\n",
      "    \"output_dir\": \"/opt/ml/output\",\n",
      "    \"output_intermediate_dir\": \"/opt/ml/output/intermediate\",\n",
      "    \"resource_config\": {\n",
      "        \"current_host\": \"algo-1\",\n",
      "        \"current_instance_type\": \"ml.c5.2xlarge\",\n",
      "        \"current_group_name\": \"homogeneousCluster\",\n",
      "        \"hosts\": [\n",
      "            \"algo-1\",\n",
      "            \"algo-2\"\n",
      "        ],\n",
      "        \"instance_groups\": [\n",
      "            {\n",
      "                \"instance_group_name\": \"homogeneousCluster\",\n",
      "                \"instance_type\": \"ml.c5.2xlarge\",\n",
      "                \"hosts\": [\n",
      "                    \"algo-2\",\n",
      "                    \"algo-1\"\n",
      "                ]\n",
      "            }\n",
      "        ],\n",
      "        \"network_interface_name\": \"eth0\"\n",
      "    },\n",
      "    \"user_entry_point\": \"mnist.py\"\u001b[0m\n",
      "\u001b[34m}\u001b[0m\n",
      "\u001b[34mEnvironment variables:\u001b[0m\n",
      "\u001b[34mSM_HOSTS=[\"algo-1\",\"algo-2\"]\u001b[0m\n",
      "\u001b[34mSM_NETWORK_INTERFACE_NAME=eth0\u001b[0m\n",
      "\u001b[34mSM_HPS={\"backend\":\"gloo\",\"epochs\":1}\u001b[0m\n",
      "\u001b[34mSM_USER_ENTRY_POINT=mnist.py\u001b[0m\n",
      "\u001b[34mSM_FRAMEWORK_PARAMS={}\u001b[0m\n",
      "\u001b[34mSM_RESOURCE_CONFIG={\"current_group_name\":\"homogeneousCluster\",\"current_host\":\"algo-1\",\"current_instance_type\":\"ml.c5.2xlarge\",\"hosts\":[\"algo-1\",\"algo-2\"],\"instance_groups\":[{\"hosts\":[\"algo-2\",\"algo-1\"],\"instance_group_name\":\"homogeneousCluster\",\"instance_type\":\"ml.c5.2xlarge\"}],\"network_interface_name\":\"eth0\"}\u001b[0m\n",
      "\u001b[34mSM_INPUT_DATA_CONFIG={\"training\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"}}\u001b[0m\n",
      "\u001b[34mSM_OUTPUT_DATA_DIR=/opt/ml/output/data\u001b[0m\n",
      "\u001b[34mSM_CHANNELS=[\"training\"]\u001b[0m\n",
      "\u001b[34mSM_CURRENT_HOST=algo-1\u001b[0m\n",
      "\u001b[34mSM_CURRENT_INSTANCE_TYPE=ml.c5.2xlarge\u001b[0m\n",
      "\u001b[34mSM_CURRENT_INSTANCE_GROUP=homogeneousCluster\u001b[0m\n",
      "\u001b[34mSM_CURRENT_INSTANCE_GROUP_HOSTS=[\"algo-2\",\"algo-1\"]\u001b[0m\n",
      "\u001b[34mSM_INSTANCE_GROUPS=[\"homogeneousCluster\"]\u001b[0m\n",
      "\u001b[34mSM_INSTANCE_GROUPS_DICT={\"homogeneousCluster\":{\"hosts\":[\"algo-2\",\"algo-1\"],\"instance_group_name\":\"homogeneousCluster\",\"instance_type\":\"ml.c5.2xlarge\"}}\u001b[0m\n",
      "\u001b[34mSM_DISTRIBUTION_INSTANCE_GROUPS=[]\u001b[0m\n",
      "\u001b[34mSM_IS_HETERO=false\u001b[0m\n",
      "\u001b[34mSM_MODULE_NAME=mnist\u001b[0m\n",
      "\u001b[34mSM_LOG_LEVEL=20\u001b[0m\n",
      "\u001b[34mSM_FRAMEWORK_MODULE=sagemaker_pytorch_container.training:main\u001b[0m\n",
      "\u001b[34mSM_INPUT_DIR=/opt/ml/input\u001b[0m\n",
      "\u001b[34mSM_INPUT_CONFIG_DIR=/opt/ml/input/config\u001b[0m\n",
      "\u001b[34mSM_OUTPUT_DIR=/opt/ml/output\u001b[0m\n",
      "\u001b[34mSM_NUM_CPUS=8\u001b[0m\n",
      "\u001b[34mSM_NUM_GPUS=0\u001b[0m\n",
      "\u001b[34mSM_MODEL_DIR=/opt/ml/model\u001b[0m\n",
      "\u001b[34mSM_MODULE_DIR=s3://sagemaker-ap-south-1-023180687239/pytorch-training-2022-10-11-07-14-12-085/source/sourcedir.tar.gz\u001b[0m\n",
      "\u001b[34mSM_TRAINING_ENV={\"additional_framework_parameters\":{},\"channel_input_dirs\":{\"training\":\"/opt/ml/input/data/training\"},\"current_host\":\"algo-1\",\"current_instance_group\":\"homogeneousCluster\",\"current_instance_group_hosts\":[\"algo-2\",\"algo-1\"],\"current_instance_type\":\"ml.c5.2xlarge\",\"distribution_hosts\":[],\"distribution_instance_groups\":[],\"framework_module\":\"sagemaker_pytorch_container.training:main\",\"hosts\":[\"algo-1\",\"algo-2\"],\"hyperparameters\":{\"backend\":\"gloo\",\"epochs\":1},\"input_config_dir\":\"/opt/ml/input/config\",\"input_data_config\":{\"training\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"}},\"input_dir\":\"/opt/ml/input\",\"instance_groups\":[\"homogeneousCluster\"],\"instance_groups_dict\":{\"homogeneousCluster\":{\"hosts\":[\"algo-2\",\"algo-1\"],\"instance_group_name\":\"homogeneousCluster\",\"instance_type\":\"ml.c5.2xlarge\"}},\"is_hetero\":false,\"is_master\":true,\"is_modelparallel_enabled\":null,\"job_name\":\"pytorch-training-2022-10-11-07-14-12-085\",\"log_level\":20,\"master_hostname\":\"algo-1\",\"model_dir\":\"/opt/ml/model\",\"module_dir\":\"s3://sagemaker-ap-south-1-023180687239/pytorch-training-2022-10-11-07-14-12-085/source/sourcedir.tar.gz\",\"module_name\":\"mnist\",\"network_interface_name\":\"eth0\",\"num_cpus\":8,\"num_gpus\":0,\"output_data_dir\":\"/opt/ml/output/data\",\"output_dir\":\"/opt/ml/output\",\"output_intermediate_dir\":\"/opt/ml/output/intermediate\",\"resource_config\":{\"current_group_name\":\"homogeneousCluster\",\"current_host\":\"algo-1\",\"current_instance_type\":\"ml.c5.2xlarge\",\"hosts\":[\"algo-1\",\"algo-2\"],\"instance_groups\":[{\"hosts\":[\"algo-2\",\"algo-1\"],\"instance_group_name\":\"homogeneousCluster\",\"instance_type\":\"ml.c5.2xlarge\"}],\"network_interface_name\":\"eth0\"},\"user_entry_point\":\"mnist.py\"}\u001b[0m\n",
      "\u001b[34mSM_USER_ARGS=[\"--backend\",\"gloo\",\"--epochs\",\"1\"]\u001b[0m\n",
      "\u001b[34mSM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate\u001b[0m\n",
      "\u001b[34mSM_CHANNEL_TRAINING=/opt/ml/input/data/training\u001b[0m\n",
      "\u001b[34mSM_HP_BACKEND=gloo\u001b[0m\n",
      "\u001b[34mSM_HP_EPOCHS=1\u001b[0m\n",
      "\u001b[34mPYTHONPATH=/opt/ml/code:/opt/conda/bin:/opt/conda/lib/python38.zip:/opt/conda/lib/python3.8:/opt/conda/lib/python3.8/lib-dynload:/opt/conda/lib/python3.8/site-packages:/opt/conda/lib/python3.8/site-packages/smdebug-1.0.22b20221003-py3.8.egg:/opt/conda/lib/python3.8/site-packages/pyinstrument-3.4.2-py3.8.egg:/opt/conda/lib/python3.8/site-packages/pyinstrument_cext-0.2.4-py3.8-linux-x86_64.egg\u001b[0m\n",
      "\u001b[34mInvoking script with the following command:\u001b[0m\n",
      "\u001b[34m/opt/conda/bin/python3.8 mnist.py --backend gloo --epochs 1\u001b[0m\n",
      "\u001b[35m2022-10-11 07:16:47,969 sagemaker-training-toolkit INFO     Waiting for the process to finish and give a return code.\u001b[0m\n",
      "\u001b[35m2022-10-11 07:16:47,969 sagemaker-training-toolkit INFO     Done waiting for a return code. Received 0 from exiting process.\u001b[0m\n",
      "\u001b[35m2022-10-11 07:16:47,973 sagemaker-training-toolkit INFO     No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[35m2022-10-11 07:16:47,983 sagemaker-training-toolkit INFO     No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[35m2022-10-11 07:16:47,993 sagemaker-training-toolkit INFO     No GPUs detected (normal if no gpus installed)\u001b[0m\n",
      "\u001b[35m2022-10-11 07:16:48,000 sagemaker-training-toolkit INFO     Invoking user script\u001b[0m\n",
      "\u001b[35mTraining Env:\u001b[0m\n",
      "\u001b[35m{\n",
      "    \"additional_framework_parameters\": {},\n",
      "    \"channel_input_dirs\": {\n",
      "        \"training\": \"/opt/ml/input/data/training\"\n",
      "    },\n",
      "    \"current_host\": \"algo-2\",\n",
      "    \"current_instance_group\": \"homogeneousCluster\",\n",
      "    \"current_instance_group_hosts\": [\n",
      "        \"algo-2\",\n",
      "        \"algo-1\"\n",
      "    ],\n",
      "    \"current_instance_type\": \"ml.c5.2xlarge\",\n",
      "    \"distribution_hosts\": [],\n",
      "    \"distribution_instance_groups\": [],\n",
      "    \"framework_module\": \"sagemaker_pytorch_container.training:main\",\n",
      "    \"hosts\": [\n",
      "        \"algo-1\",\n",
      "        \"algo-2\"\n",
      "    ],\n",
      "    \"hyperparameters\": {\n",
      "        \"backend\": \"gloo\",\n",
      "        \"epochs\": 1\n",
      "    },\n",
      "    \"input_config_dir\": \"/opt/ml/input/config\",\n",
      "    \"input_data_config\": {\n",
      "        \"training\": {\n",
      "            \"TrainingInputMode\": \"File\",\n",
      "            \"S3DistributionType\": \"FullyReplicated\",\n",
      "            \"RecordWrapperType\": \"None\"\n",
      "        }\n",
      "    },\n",
      "    \"input_dir\": \"/opt/ml/input\",\n",
      "    \"instance_groups\": [\n",
      "        \"homogeneousCluster\"\n",
      "    ],\n",
      "    \"instance_groups_dict\": {\n",
      "        \"homogeneousCluster\": {\n",
      "            \"instance_group_name\": \"homogeneousCluster\",\n",
      "            \"instance_type\": \"ml.c5.2xlarge\",\n",
      "            \"hosts\": [\n",
      "                \"algo-2\",\n",
      "                \"algo-1\"\n",
      "            ]\n",
      "        }\n",
      "    },\n",
      "    \"is_hetero\": false,\n",
      "    \"is_master\": false,\n",
      "    \"is_modelparallel_enabled\": null,\n",
      "    \"job_name\": \"pytorch-training-2022-10-11-07-14-12-085\",\n",
      "    \"log_level\": 20,\n",
      "    \"master_hostname\": \"algo-1\",\n",
      "    \"model_dir\": \"/opt/ml/model\",\n",
      "    \"module_dir\": \"s3://sagemaker-ap-south-1-023180687239/pytorch-training-2022-10-11-07-14-12-085/source/sourcedir.tar.gz\",\n",
      "    \"module_name\": \"mnist\",\n",
      "    \"network_interface_name\": \"eth0\",\n",
      "    \"num_cpus\": 8,\n",
      "    \"num_gpus\": 0,\n",
      "    \"output_data_dir\": \"/opt/ml/output/data\",\n",
      "    \"output_dir\": \"/opt/ml/output\",\n",
      "    \"output_intermediate_dir\": \"/opt/ml/output/intermediate\",\n",
      "    \"resource_config\": {\n",
      "        \"current_host\": \"algo-2\",\n",
      "        \"current_instance_type\": \"ml.c5.2xlarge\",\n",
      "        \"current_group_name\": \"homogeneousCluster\",\n",
      "        \"hosts\": [\n",
      "            \"algo-1\",\n",
      "            \"algo-2\"\n",
      "        ],\n",
      "        \"instance_groups\": [\n",
      "            {\n",
      "                \"instance_group_name\": \"homogeneousCluster\",\n",
      "                \"instance_type\": \"ml.c5.2xlarge\",\n",
      "                \"hosts\": [\n",
      "                    \"algo-2\",\n",
      "                    \"algo-1\"\n",
      "                ]\n",
      "            }\n",
      "        ],\n",
      "        \"network_interface_name\": \"eth0\"\n",
      "    },\n",
      "    \"user_entry_point\": \"mnist.py\"\u001b[0m\n",
      "\u001b[35m}\u001b[0m\n",
      "\u001b[35mEnvironment variables:\u001b[0m\n",
      "\u001b[35mSM_HOSTS=[\"algo-1\",\"algo-2\"]\u001b[0m\n",
      "\u001b[35mSM_NETWORK_INTERFACE_NAME=eth0\u001b[0m\n",
      "\u001b[35mSM_HPS={\"backend\":\"gloo\",\"epochs\":1}\u001b[0m\n",
      "\u001b[35mSM_USER_ENTRY_POINT=mnist.py\u001b[0m\n",
      "\u001b[35mSM_FRAMEWORK_PARAMS={}\u001b[0m\n",
      "\u001b[35mSM_RESOURCE_CONFIG={\"current_group_name\":\"homogeneousCluster\",\"current_host\":\"algo-2\",\"current_instance_type\":\"ml.c5.2xlarge\",\"hosts\":[\"algo-1\",\"algo-2\"],\"instance_groups\":[{\"hosts\":[\"algo-2\",\"algo-1\"],\"instance_group_name\":\"homogeneousCluster\",\"instance_type\":\"ml.c5.2xlarge\"}],\"network_interface_name\":\"eth0\"}\u001b[0m\n",
      "\u001b[35mSM_INPUT_DATA_CONFIG={\"training\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"}}\u001b[0m\n",
      "\u001b[35mSM_OUTPUT_DATA_DIR=/opt/ml/output/data\u001b[0m\n",
      "\u001b[35mSM_CHANNELS=[\"training\"]\u001b[0m\n",
      "\u001b[35mSM_CURRENT_HOST=algo-2\u001b[0m\n",
      "\u001b[35mSM_CURRENT_INSTANCE_TYPE=ml.c5.2xlarge\u001b[0m\n",
      "\u001b[35mSM_CURRENT_INSTANCE_GROUP=homogeneousCluster\u001b[0m\n",
      "\u001b[35mSM_CURRENT_INSTANCE_GROUP_HOSTS=[\"algo-2\",\"algo-1\"]\u001b[0m\n",
      "\u001b[35mSM_INSTANCE_GROUPS=[\"homogeneousCluster\"]\u001b[0m\n",
      "\u001b[35mSM_INSTANCE_GROUPS_DICT={\"homogeneousCluster\":{\"hosts\":[\"algo-2\",\"algo-1\"],\"instance_group_name\":\"homogeneousCluster\",\"instance_type\":\"ml.c5.2xlarge\"}}\u001b[0m\n",
      "\u001b[35mSM_DISTRIBUTION_INSTANCE_GROUPS=[]\u001b[0m\n",
      "\u001b[35mSM_IS_HETERO=false\u001b[0m\n",
      "\u001b[35mSM_MODULE_NAME=mnist\u001b[0m\n",
      "\u001b[35mSM_LOG_LEVEL=20\u001b[0m\n",
      "\u001b[35mSM_FRAMEWORK_MODULE=sagemaker_pytorch_container.training:main\u001b[0m\n",
      "\u001b[35mSM_INPUT_DIR=/opt/ml/input\u001b[0m\n",
      "\u001b[35mSM_INPUT_CONFIG_DIR=/opt/ml/input/config\u001b[0m\n",
      "\u001b[35mSM_OUTPUT_DIR=/opt/ml/output\u001b[0m\n",
      "\u001b[35mSM_NUM_CPUS=8\u001b[0m\n",
      "\u001b[35mSM_NUM_GPUS=0\u001b[0m\n",
      "\u001b[35mSM_MODEL_DIR=/opt/ml/model\u001b[0m\n",
      "\u001b[35mSM_MODULE_DIR=s3://sagemaker-ap-south-1-023180687239/pytorch-training-2022-10-11-07-14-12-085/source/sourcedir.tar.gz\u001b[0m\n",
      "\u001b[35mSM_TRAINING_ENV={\"additional_framework_parameters\":{},\"channel_input_dirs\":{\"training\":\"/opt/ml/input/data/training\"},\"current_host\":\"algo-2\",\"current_instance_group\":\"homogeneousCluster\",\"current_instance_group_hosts\":[\"algo-2\",\"algo-1\"],\"current_instance_type\":\"ml.c5.2xlarge\",\"distribution_hosts\":[],\"distribution_instance_groups\":[],\"framework_module\":\"sagemaker_pytorch_container.training:main\",\"hosts\":[\"algo-1\",\"algo-2\"],\"hyperparameters\":{\"backend\":\"gloo\",\"epochs\":1},\"input_config_dir\":\"/opt/ml/input/config\",\"input_data_config\":{\"training\":{\"RecordWrapperType\":\"None\",\"S3DistributionType\":\"FullyReplicated\",\"TrainingInputMode\":\"File\"}},\"input_dir\":\"/opt/ml/input\",\"instance_groups\":[\"homogeneousCluster\"],\"instance_groups_dict\":{\"homogeneousCluster\":{\"hosts\":[\"algo-2\",\"algo-1\"],\"instance_group_name\":\"homogeneousCluster\",\"instance_type\":\"ml.c5.2xlarge\"}},\"is_hetero\":false,\"is_master\":false,\"is_modelparallel_enabled\":null,\"job_name\":\"pytorch-training-2022-10-11-07-14-12-085\",\"log_level\":20,\"master_hostname\":\"algo-1\",\"model_dir\":\"/opt/ml/model\",\"module_dir\":\"s3://sagemaker-ap-south-1-023180687239/pytorch-training-2022-10-11-07-14-12-085/source/sourcedir.tar.gz\",\"module_name\":\"mnist\",\"network_interface_name\":\"eth0\",\"num_cpus\":8,\"num_gpus\":0,\"output_data_dir\":\"/opt/ml/output/data\",\"output_dir\":\"/opt/ml/output\",\"output_intermediate_dir\":\"/opt/ml/output/intermediate\",\"resource_config\":{\"current_group_name\":\"homogeneousCluster\",\"current_host\":\"algo-2\",\"current_instance_type\":\"ml.c5.2xlarge\",\"hosts\":[\"algo-1\",\"algo-2\"],\"instance_groups\":[{\"hosts\":[\"algo-2\",\"algo-1\"],\"instance_group_name\":\"homogeneousCluster\",\"instance_type\":\"ml.c5.2xlarge\"}],\"network_interface_name\":\"eth0\"},\"user_entry_point\":\"mnist.py\"}\u001b[0m\n",
      "\u001b[35mSM_USER_ARGS=[\"--backend\",\"gloo\",\"--epochs\",\"1\"]\u001b[0m\n",
      "\u001b[35mSM_OUTPUT_INTERMEDIATE_DIR=/opt/ml/output/intermediate\u001b[0m\n",
      "\u001b[35mSM_CHANNEL_TRAINING=/opt/ml/input/data/training\u001b[0m\n",
      "\u001b[35mSM_HP_BACKEND=gloo\u001b[0m\n",
      "\u001b[35mSM_HP_EPOCHS=1\u001b[0m\n",
      "\u001b[35mPYTHONPATH=/opt/ml/code:/opt/conda/bin:/opt/conda/lib/python38.zip:/opt/conda/lib/python3.8:/opt/conda/lib/python3.8/lib-dynload:/opt/conda/lib/python3.8/site-packages:/opt/conda/lib/python3.8/site-packages/smdebug-1.0.22b20221003-py3.8.egg:/opt/conda/lib/python3.8/site-packages/pyinstrument-3.4.2-py3.8.egg:/opt/conda/lib/python3.8/site-packages/pyinstrument_cext-0.2.4-py3.8-linux-x86_64.egg\u001b[0m\n",
      "\u001b[35mInvoking script with the following command:\u001b[0m\n",
      "\u001b[35m/opt/conda/bin/python3.8 mnist.py --backend gloo --epochs 1\u001b[0m\n",
      "\u001b[34mDistributed training - True\u001b[0m\n",
      "\u001b[34mNumber of gpus available - 0\u001b[0m\n",
      "\u001b[34mInitialized the distributed environment: 'gloo' backend on 2 nodes. Current host rank is 0. Number of gpus: 0\u001b[0m\n",
      "\u001b[34mGet train data loader\u001b[0m\n",
      "\u001b[34mGet test data loader\u001b[0m\n",
      "\u001b[34mProcesses 30000/60000 (50%) of train data\u001b[0m\n",
      "\u001b[34mProcesses 10000/10000 (100%) of test data\u001b[0m\n",
      "\u001b[35mDistributed training - True\u001b[0m\n",
      "\u001b[35mNumber of gpus available - 0\u001b[0m\n",
      "\u001b[35mInitialized the distributed environment: 'gloo' backend on 2 nodes. Current host rank is 1. Number of gpus: 0\u001b[0m\n",
      "\u001b[35mGet train data loader\u001b[0m\n",
      "\u001b[35mGet test data loader\u001b[0m\n",
      "\u001b[35mProcesses 30000/60000 (50%) of train data\u001b[0m\n",
      "\u001b[35mProcesses 10000/10000 (100%) of test data\u001b[0m\n",
      "\u001b[35m[2022-10-11 07:16:50.640 algo-2:44 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[2022-10-11 07:16:50.640 algo-1:45 INFO utils.py:27] RULE_JOB_STOP_SIGNAL_FILENAME: None\u001b[0m\n",
      "\u001b[34m[2022-10-11 07:16:51.004 algo-1:45 INFO profiler_config_parser.py:111] User has disabled profiler.\u001b[0m\n",
      "\u001b[34m[2022-10-11 07:16:51.005 algo-1:45 INFO json_config.py:91] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[34m[2022-10-11 07:16:51.005 algo-1:45 INFO hook.py:201] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[34m[2022-10-11 07:16:51.006 algo-1:45 INFO hook.py:254] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[34m[2022-10-11 07:16:51.006 algo-1:45 INFO state_store.py:77] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[34m/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:181: UserWarning: torch.distributed.reduce_op is deprecated, please use torch.distributed.ReduceOp instead\n",
      "  warnings.warn(\u001b[0m\n",
      "\u001b[35m[2022-10-11 07:16:51.079 algo-2:44 INFO profiler_config_parser.py:111] User has disabled profiler.\u001b[0m\n",
      "\u001b[35m[2022-10-11 07:16:51.080 algo-2:44 INFO json_config.py:91] Creating hook from json_config at /opt/ml/input/config/debughookconfig.json.\u001b[0m\n",
      "\u001b[35m[2022-10-11 07:16:51.081 algo-2:44 INFO hook.py:201] tensorboard_dir has not been set for the hook. SMDebug will not be exporting tensorboard summaries.\u001b[0m\n",
      "\u001b[35m[2022-10-11 07:16:51.081 algo-2:44 INFO hook.py:254] Saving to /opt/ml/output/tensors\u001b[0m\n",
      "\u001b[35m[2022-10-11 07:16:51.081 algo-2:44 INFO state_store.py:77] The checkpoint config file /opt/ml/input/config/checkpointconfig.json does not exist.\u001b[0m\n",
      "\u001b[35m/opt/conda/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py:181: UserWarning: torch.distributed.reduce_op is deprecated, please use torch.distributed.ReduceOp instead\n",
      "  warnings.warn(\u001b[0m\n",
      "\u001b[34mTrain Epoch: 1 [6400/30000 (21%)] Loss: 2.076445\u001b[0m\n",
      "\u001b[34mINFO:__main__:Train Epoch: 1 [6400/30000 (21%)] Loss: 2.076445\u001b[0m\n",
      "\u001b[35mTrain Epoch: 1 [6400/30000 (21%)] Loss: 2.075126\u001b[0m\n",
      "\u001b[35mINFO:__main__:Train Epoch: 1 [6400/30000 (21%)] Loss: 2.075126\u001b[0m\n",
      "\u001b[34mTrain Epoch: 1 [12800/30000 (43%)] Loss: 1.056658\u001b[0m\n",
      "\u001b[34mINFO:__main__:Train Epoch: 1 [12800/30000 (43%)] Loss: 1.056658\u001b[0m\n",
      "\u001b[35mTrain Epoch: 1 [12800/30000 (43%)] Loss: 1.213364\u001b[0m\n",
      "\u001b[35mINFO:__main__:Train Epoch: 1 [12800/30000 (43%)] Loss: 1.213364\u001b[0m\n",
      "\u001b[35mTrain Epoch: 1 [19200/30000 (64%)] Loss: 0.908578\u001b[0m\n",
      "\u001b[35mINFO:__main__:Train Epoch: 1 [19200/30000 (64%)] Loss: 0.908578\u001b[0m\n",
      "\u001b[34mTrain Epoch: 1 [19200/30000 (64%)] Loss: 0.944071\u001b[0m\n",
      "\u001b[34mINFO:__main__:Train Epoch: 1 [19200/30000 (64%)] Loss: 0.944071\u001b[0m\n",
      "\u001b[35mTrain Epoch: 1 [25600/30000 (85%)] Loss: 0.671730\u001b[0m\n",
      "\u001b[35mINFO:__main__:Train Epoch: 1 [25600/30000 (85%)] Loss: 0.671730\u001b[0m\n",
      "\u001b[35m/opt/conda/lib/python3.8/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.\n",
      "  warnings.warn(warning.format(ret))\u001b[0m\n",
      "\n",
      "2022-10-11 07:17:08 Uploading - Uploading generated training model\u001b[34mTrain Epoch: 1 [25600/30000 (85%)] Loss: 0.846429\u001b[0m\n",
      "\u001b[34mINFO:__main__:Train Epoch: 1 [25600/30000 (85%)] Loss: 0.846429\u001b[0m\n",
      "\u001b[34m/opt/conda/lib/python3.8/site-packages/torch/nn/_reduction.py:42: UserWarning: size_average and reduce args will be deprecated, please use reduction='sum' instead.\n",
      "  warnings.warn(warning.format(ret))\u001b[0m\n",
      "\u001b[34mTest set: Average loss: 0.3238, Accuracy: 9094/10000 (91%)\u001b[0m\n",
      "\u001b[34mSaving the model.\u001b[0m\n",
      "\u001b[34mINFO:__main__:Test set: Average loss: 0.3238, Accuracy: 9094/10000 (91%)\u001b[0m\n",
      "\u001b[34mINFO:__main__:Saving the model.\u001b[0m\n",
      "\u001b[35mTest set: Average loss: 0.3238, Accuracy: 9094/10000 (91%)\u001b[0m\n",
      "\u001b[35mSaving the model.\u001b[0m\n",
      "\u001b[35mINFO:__main__:Test set: Average loss: 0.3238, Accuracy: 9094/10000 (91%)\u001b[0m\n",
      "\u001b[35mINFO:__main__:Saving the model.\u001b[0m\n",
      "\u001b[35m2022-10-11 07:17:01,626 sagemaker-training-toolkit INFO     Waiting for the process to finish and give a return code.\u001b[0m\n",
      "\u001b[35m2022-10-11 07:17:01,627 sagemaker-training-toolkit INFO     Done waiting for a return code. Received 0 from exiting process.\u001b[0m\n",
      "\u001b[35m2022-10-11 07:17:01,627 sagemaker-training-toolkit INFO     Reporting training SUCCESS\u001b[0m\n",
      "\u001b[34m2022-10-11 07:17:01,783 sagemaker-training-toolkit INFO     Waiting for the process to finish and give a return code.\u001b[0m\n",
      "\u001b[34m2022-10-11 07:17:01,783 sagemaker-training-toolkit INFO     Done waiting for a return code. Received 0 from exiting process.\u001b[0m\n",
      "\u001b[34m2022-10-11 07:17:01,783 sagemaker-training-toolkit INFO     Reporting training SUCCESS\u001b[0m\n",
      "\n",
      "2022-10-11 07:17:37 Completed - Training job completed\n",
      "Training seconds: 224\n",
      "Billable seconds: 224\n"
     ]
    }
   ],
   "source": [
    "estimator.fit({\"training\": inputs})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Host\n",
    "### Create endpoint\n",
    "After training, we use the `PyTorch` estimator object to build and deploy a `PyTorchPredictor`. This creates a Sagemaker Endpoint -- a hosted prediction service that we can use to perform inference.\n",
    "\n",
    "As mentioned above we have implementation of `model_fn` in the `mnist.py` script that is required. We are going to use default implementations of `input_fn`, `predict_fn`, `output_fn` and `transform_fm` defined in [sagemaker-pytorch-containers](https://github.com/aws/sagemaker-pytorch-containers).\n",
    "\n",
    "The arguments to the deploy function allow us to set the number and type of instances that will be used for the Endpoint. These do not need to be the same as the values we used for the training job. For example, you can train a model on a set of GPU-based instances, and then deploy the Endpoint to a fleet of CPU-based instances, but you need to make sure that you return or save your model as a cpu model similar to what we did in `mnist.py`. Here we will deploy the model to a single ```ml.m4.xlarge``` instance."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'framework_version': '1.11.0',\n",
       " 'py_version': 'py38',\n",
       " 'role': 'arn:aws:iam::023180687239:role/service-role/AmazonSageMaker-ExecutionRole-20220906T142944',\n",
       " 'instance_count': 2,\n",
       " 'instance_type': 'ml.c5.2xlarge',\n",
       " 'keep_alive_period_in_seconds': None,\n",
       " 'instance_groups': None,\n",
       " 'volume_size': 30,\n",
       " 'volume_kms_key': None,\n",
       " 'max_run': 86400,\n",
       " 'input_mode': 'File',\n",
       " 'metric_definitions': None,\n",
       " 'model_uri': None,\n",
       " 'model_channel_name': 'model',\n",
       " 'code_uri': None,\n",
       " 'code_channel_name': 'code',\n",
       " 'source_dir': 'code',\n",
       " 'git_config': None,\n",
       " 'container_log_level': 20,\n",
       " '_hyperparameters': {'epochs': 1,\n",
       "  'backend': 'gloo',\n",
       "  'sagemaker_submit_directory': 's3://sagemaker-ap-south-1-023180687239/pytorch-training-2022-10-11-07-14-12-085/source/sourcedir.tar.gz',\n",
       "  'sagemaker_program': 'mnist.py',\n",
       "  'sagemaker_container_log_level': 20,\n",
       "  'sagemaker_job_name': 'pytorch-training-2022-10-11-07-14-12-085',\n",
       "  'sagemaker_region': 'ap-south-1'},\n",
       " 'code_location': None,\n",
       " 'entry_point': 'mnist.py',\n",
       " 'dependencies': ['code/requirements.txt'],\n",
       " 'uploaded_code': UserCode(s3_prefix='s3://sagemaker-ap-south-1-023180687239/pytorch-training-2022-10-11-07-14-12-085/source/sourcedir.tar.gz', script_name='mnist.py'),\n",
       " 'tags': None,\n",
       " 'sagemaker_session': <sagemaker.session.Session at 0x7efc9892cd50>,\n",
       " 'base_job_name': 'pytorch-training',\n",
       " '_current_job_name': 'pytorch-training-2022-10-11-07-14-12-085',\n",
       " 'output_path': 's3://sagemaker-ap-south-1-023180687239/',\n",
       " 'output_kms_key': None,\n",
       " 'latest_training_job': <sagemaker.estimator._TrainingJob at 0x7efc99934590>,\n",
       " 'jobs': [<sagemaker.estimator._TrainingJob at 0x7efc99934590>],\n",
       " 'deploy_instance_type': None,\n",
       " '_compiled_models': {},\n",
       " 'subnets': None,\n",
       " 'security_group_ids': None,\n",
       " 'encrypt_inter_container_traffic': False,\n",
       " 'use_spot_instances': False,\n",
       " 'max_wait': None,\n",
       " 'checkpoint_s3_uri': None,\n",
       " 'checkpoint_local_path': None,\n",
       " 'rules': None,\n",
       " 'debugger_hook_config': <sagemaker.debugger.debugger.DebuggerHookConfig at 0x7efc984e0510>,\n",
       " 'tensorboard_output_config': None,\n",
       " 'debugger_rule_configs': [],\n",
       " 'collection_configs': set(),\n",
       " 'enable_sagemaker_metrics': True,\n",
       " '_enable_network_isolation': False,\n",
       " 'profiler_config': <sagemaker.debugger.profiler_config.ProfilerConfig at 0x7efc984a1190>,\n",
       " 'disable_profiler': False,\n",
       " 'environment': None,\n",
       " 'max_retry_attempts': None,\n",
       " 'profiler_rule_configs': [{'RuleConfigurationName': 'ProfilerReport-1665472452',\n",
       "   'RuleEvaluatorImage': '904829902805.dkr.ecr.ap-south-1.amazonaws.com/sagemaker-debugger-rules:latest',\n",
       "   'RuleParameters': {'rule_to_invoke': 'ProfilerReport'}}],\n",
       " 'profiler_rules': [ProfilerRule(name='ProfilerReport-1665472452', image_uri='904829902805.dkr.ecr.ap-south-1.amazonaws.com/sagemaker-debugger-rules:latest', instance_type=None, container_local_output_path=None, s3_output_path=None, volume_size_in_gb=None, rule_parameters={'rule_to_invoke': 'ProfilerReport'})],\n",
       " 'debugger_rules': [],\n",
       " 'image_uri': None,\n",
       " 'distribution': {}}"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "estimator.__dict__\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-----!"
     ]
    }
   ],
   "source": [
    "predictor = estimator.deploy(initial_instance_count=1, instance_type=\"ml.c5.2xlarge\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Evaluate\n",
    "\n",
    "You can use the test images to evalute the endpoint. The accuracy of the model depends on how many it is trained. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "### Cleanup\n",
    "\n",
    "After you have finished with this example, remember to delete the prediction endpoint to release the instance(s) associated with it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sagemaker_session.delete_endpoint(endpoint_name=predictor.endpoint_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'pytorch-training-2022-10-11-08-07-06-661'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictor.endpoint_name "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_url = \"\" #add url of sample test data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'ResponseMetadata': {'RequestId': '0957ab6d-569b-477a-aef0-7481a935eecb', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amzn-requestid': '0957ab6d-569b-477a-aef0-7481a935eecb', 'x-amzn-invoked-production-variant': 'AllTraffic', 'date': 'Tue, 11 Oct 2022 08:23:34 GMT', 'content-type': 'application/json', 'content-length': '207'}, 'RetryAttempts': 0}, 'ContentType': 'application/json', 'InvokedProductionVariant': 'AllTraffic', 'Body': <botocore.response.StreamingBody object at 0x7efc7f467550>}\n"
     ]
    }
   ],
   "source": [
    "import boto3\n",
    "\n",
    "client = boto3.client('sagemaker-runtime')\n",
    "\n",
    "custom_attributes = \"c000b4f9-df62-4c85-a0bf-7c525f9104a4\"  # An example of a trace ID.\n",
    "endpoint_name = predictor.endpoint_name                                        # Your endpoint name.\n",
    "content_type = \"application/json\"                                        # The MIME type of the input data in the request body.\n",
    "accept = \"application/json\"                                              # The desired MIME type of the inference in the response.\n",
    "payload = json.dumps({\"url\":data_url})                                           # Payload for inference.\n",
    "response = client.invoke_endpoint(\n",
    "    EndpointName=endpoint_name, \n",
    "    CustomAttributes=custom_attributes, \n",
    "    ContentType=content_type,\n",
    "    Accept=accept,\n",
    "    Body=payload\n",
    "    )\n",
    "\n",
    "print(response)   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pprint import pprint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'Body': <botocore.response.StreamingBody object at 0x7efc7f467550>,\n",
      " 'ContentType': 'application/json',\n",
      " 'InvokedProductionVariant': 'AllTraffic',\n",
      " 'ResponseMetadata': {'HTTPHeaders': {'content-length': '207',\n",
      "                                      'content-type': 'application/json',\n",
      "                                      'date': 'Tue, 11 Oct 2022 08:23:34 GMT',\n",
      "                                      'x-amzn-invoked-production-variant': 'AllTraffic',\n",
      "                                      'x-amzn-requestid': '0957ab6d-569b-477a-aef0-7481a935eecb'},\n",
      "                      'HTTPStatusCode': 200,\n",
      "                      'RequestId': '0957ab6d-569b-477a-aef0-7481a935eecb',\n",
      "                      'RetryAttempts': 0}}\n"
     ]
    }
   ],
   "source": [
    "pprint(response) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "r = json.load(response[\"Body\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[-3.0102062225341797,\n",
       "  -2.48018217086792,\n",
       "  -1.5776013135910034,\n",
       "  -2.6531476974487305,\n",
       "  -3.039762020111084,\n",
       "  -1.8981196880340576,\n",
       "  -2.262784481048584,\n",
       "  -4.358467102050781,\n",
       "  -1.3877885341644287,\n",
       "  -3.6531260013580322]]"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3.8.8 ('base')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "notice": "Copyright 2018 Amazon.com, Inc. or its affiliates. All Rights Reserved.  Licensed under the Apache License, Version 2.0 (the \"License\"). You may not use this file except in compliance with the License. A copy of the License is located at http://aws.amazon.com/apache2.0/ or in the \"license\" file accompanying this file. This file is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.",
  "vscode": {
   "interpreter": {
    "hash": "2ed236227034f4f34b7af70d7e301e7e416b9eea7e7665e4e660cbbba097d900"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}