Tracking_GPU_Memory_Usage.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "Tracking_GPU_Memory_Usage.ipynb",
      "provenance": [],
      "collapsed_sections": [
        "7mnq8h-hG16N",
        "_FgrIBGIPTYT",
        "dk2AGJUBNQab",
        "q_9cheI9b-1w",
        "TJh0jGe2Ncl6",
        "bBjQsif-c1oQ",
        "n6LQkC3NbPM3",
        "H6pP9jJBPjcj"
      ],
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/kannankumar/jupyter-notebooks/blob/master/Tracking_GPU_Memory_Usage.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aW8eE3Qwb5xP",
        "colab_type": "text"
      },
      "source": [
        "# Tracking GPU Memory Usage\n",
        "\n",
        "The most amazing thing about Collaboratory (or Google's generousity) is that there's also GPU option available.  \n",
        "\n",
        "In this short notebook we look at how to track GPU memory usage.\n",
        "\n",
        "This notebook has been divided into sections.\n",
        "Feel free to skip the sections which you are already familar with.\n",
        "\n",
        "-----\n",
        "**Footnote**: *This notebook is a fork of [a great Colab notebook](https://colab.research.google.com/drive/1P7okDVh6viCIOkii6UAF2O9sTAcKGNWq) and it is edited for my personal reference.*\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7mnq8h-hG16N",
        "colab_type": "text"
      },
      "source": [
        "## Enabling GPU (on Colab)\n",
        "If you are using a Colab environment and have not tried switching to the GPU mode on the Colab notebook before, here's a quick refresher on that.  \n",
        "\n",
        "**When using another notebook/environment you will need to find out  \n",
        "how to connect to a GPU runtime on your own.**  \n",
        "\n",
        "*Sorry, I haven't perfected my environment-sensing and mind-reading skills yet.*"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oIg04gr6dpRc",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "Follow on the collaboratory menu tabs, \"Runtime\" => \"Change runtime type\".\n",
        "\n",
        "<img src=\"http://deeplearnphysics.org/Blog/imgs/2018-03-02-Collab-00-img00.png\" width=\"50%\">\n",
        "  \n",
        "\n",
        "### Choosing Runtime type\n",
        "Then you should see a pop-up where you can choose GPU.\n",
        "\n",
        "<img src=\"http://deeplearnphysics.org/Blog/imgs/2018-03-02-Collab-00-img01.png\" width=\"30%\">\n",
        "\n",
        "After you change your runtime, your runtime should automatically restart (which means information from executed cells disappear).\n",
        "  \n",
        "\n",
        "### Checking Runtime type\n",
        "A quick way to check your current runtime is to hover on the toolbar where it shows the `RAM` and `Disk` details.  \n",
        "If it mentions **`\"(GPU)\"`**, then the Colab notebook is connected to a GPU runtime. Otherwise a standard CPU runtime.\n",
        "\n",
        "<img src=\"data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAlgAAADYCAYAAAA3SZ0tAAAgAElEQVR4Ae2dBbMkx5m157et17D2GjZMu3LYXm941yA7HGJmlkbSCC1Gi8FjkcXMGo0YRkwWo8Uwqi+e/Hyu385bVZ3Vt7pvw8mI7qquSjx58s2TUNVrKjsjYASMgBEwAkbACBiBXhFY02tsjswIGAEjYASMgBEwAkagssAyCYyAETACRsAIGAEj0DMCFlg9A+rojIARMAJGwAgYASNggWUOGAEjYASMgBEwAkagZwQssHoG1NEZASNgBIyAETACRsACyxwwAkbACBgBI2AEjEDPCFhg9QyoozMCRsAIGAEjYASMgAWWOWAEjIARMAJGwAgYgZ4RsMDqGVBHZwSMgBEwAkbACBgBCyxzwAgYASNgBIyAETACPSNggdUzoI7OCBgBI2AEjIARMAIWWOaAETACRsAIGAEjYAR6RsACq2dAHZ0RMAJGwAgYASNgBCywzAEjYASMgBEwAkbACPSMgAVWz4A6OiNgBIyAETACRsAIWGCZA0bACBgBI2AEjIAR6BkBC6yeAXV0RsAIGAEjYASMgBGwwDIHjIARMAJGwAgYASPQMwIWWD0D6uiMgBEwAkbACBgBI2CBZQ4YASNgBIyAETACRqBnBCywegbU0RkBI2AEjIARMAJGYM0TTzxR+WMMpo0Dzz33XOWPMTAHRuPAtLVn58d9zCJywALLAnMqBbY71tE6VuNm3ODAInZmLrNF3LRxwEuEnsWcSgQsFCwUzIHROTCVjdqZMgILhoAF1oJV+KwU153r6J2rsTN2s9LOnU8jMM8IWGDNc+3OcNksEiwSzIHROTDDTd9ZNwJzg4AF1txU5XwVxJ3r6J2rsTN282UNXBojMJsIWGDNZr3Nfa4tEiwSzIHROTD3BsIFNAIzgIAF1gxU0iJm0Z3r6J2rsTN2i2gzXGYjMG0IWGBNW404PwkBiwSLBHNgdA7YjBgBI7D6CFhgrX4dOAc1CLhzHb1zNXbGrqZJ+ZIRMAITRsACa8KAO7kyBBZJJFxwwQXV+eefP/C55JJLqjvuuKN66qmnit5of/fdd6fwt99+e6t/4lRajzzySKtf1cH69etTmMsvv7zIv8LVHa+77rrKn3IM6jAsuVbWymbHl14gOTs5dk6NQFVZYJkFU4lASScyL36++tWvVv/yL/9S+/n2t79dHXvssUOF1vbbb5/C//a3v20VQcSltI4++uhWv+CLYJP///mf/xnqf1idWFyViyuwGoZn0/2pbNQryJQF1grAc9BVQ8ACa9Wgd8JtCDR1HPN4XQLr3HPPrS666KL0OfHEEyvE0r/+678mgXPAAQc0drYPP/xw9bWvfS35+8pXvlLdc889jX6jwPrRj35UPfvss41+wZp0xyGwTjnllCp+5rFeV1ImCdFR42hrW7N4zwJrFmvNebbAMgemEoFRO5ZZDCeB9fjjjy8TO8wgfeMb30gi584771x2n/KedNJJ6f6WW26ZjocffnitP/xKYEmQXXrppY1+n3zyyYoZtK9//esp3j5nsKK44nwW622cebbAGjRLFliDePjXbCBggTUb9bRwuRxn5zVtcbcJLPK67bbbJoFz3nnn1QqRn//852kG67777qu+9a1vVT/4wQ8aZ6YksHbYYYcUJ3E34XH22WcnPzvttJMF1nOT3ThvgTVo8iywBvHwr9lAwAJrNupp4XLZ1OnP4/VhAmvPPfdMAuf0009fJoZuuummdG+bbbZJ9+S3aUO6BBazRiwRsgT5wAMPLIsXnH/1q1+luImLZULPYE1OZFlgDZo8C6xBPPxrNhCwwJqNelq4XM6jkGoqU5vAevrpp6stttiiYm8VM1R5HPvtt18SP+zd4t7VV1+dfu+4447L/HJfAuuEE06ojjnmmOT3qKOOWuZXm9v/93//t7r33nstsDyDtao2yAJrVeF34iMiYIE1InAONl4EciExz7/rBBavZ+BVDVtttVUSN8wm5Rjghz1S//7v/770lCGb1pmZYo9V3WsYJLAQVw8++GBF2j/84Q+XLSlqc/tZZ51VsffLM1iTm72inj2DNWhfLLAG8fCv2UDAAms26mnhcpmLiXn+LYGFMPrP//zP9PnmN7+ZRA3Chv1Sjz322DKBxfusuL/XXnsN3GOTO9dPPvnkgetgKIG1bt26dE97seJmd21uR7zRsd16660WWJ7BWlUbtAgC64033qh45xxHu/lAwAJrPupx7koxz4IqL5sE1i9+8Yu0z4m9Tv/1X/+VRM2vf/3rZSJJ4X//+98nP9dcc82AH804/fd///fA9SiwDjnkkHTvyiuvTHHEze7a3K5XQ9xwww0WWBZYq2pj5l1gPf/88xVPAWMDOPLbbvYRsMCa/TqcyxJIRCzCUQIrvqZh06ZN1fe+97209+rmm29eJpQ2btyY7rE3a7vttqt40Wj86DUMeVjNYB100EFLcbLHi83u999/f7qmze289R38Wa7yEqGXCFfT0MyzwIriCoFlkbWaTOs3bQusfvF0bD0hsAjCSmWsE1jc0/utfve73y2JIYVhYzqihw8iK//o3v777z8Qtk5gHX/88Ske4rztttvS+W9+85ulcBZYkxVX1LH3YA0aknkVWB9++GF6obCEVTx6JmuQA7P4ywJrFmttAfIsITELx4sfWb8kRkbJb5PAYhM7+7IQS2x4V9xsZP/xj3+crt91111L13Wf40MPPZRmpb7zne8sbYDnep3AYjM8LxNls/u+++6b4o3v3LLAssBabZMzrwILXNnjGIVVPEdkeU/WarNv9PQtsEbHziHHiEAUC9N8vnbjumqLW35Rrd14RK3QKcl7k8AiLH+fg8D66U9/Wj3zzDMpDe2b+uUvf9ma5tZbb53C8mfSykedwOLerrvumvyyVPjd7353QJRZYFlgjbGpF0U9bwKLZUGElVybyGLj+zQ53pu3zz77VIcddtiybH388cfV5s2bl10vufDRRx+VeJspPxZYM1Vdi5NZCYJpPkpcIbD4jDqT1SawmK1iszoiSy8alRiqe0ow4nXhhRemcH/4wx+GCqxrr702+SUdbYBXXOMQWFoC87H9z59VB12P82Yp5klgxT1Xw0QW/0c66gzW3XffXfHi4ZIPr20pdRs2bEj7Pvfee++BIMy4868PBx54YPXFF18M3Bv2g/9eZQ8pD+jMk7PAmqfanKOydO1QJu0/F1f8HjUPbQKLOHmFAsLnP/7jP9K7q/hvQsLwJ89taWLweEdWfElp0wwW8fzsZz9LfjHMMV4LrHYRNE6RGOuhy/kcmYJUlHkRWFFcaSmwSWQhrvA/qmM/JQ/AlHwYVJW6JoHFn8yTFi85zmejsEX8I8Qtt9xSmwwvTCbsZZddVnt/Vi9aYM1qzc15vrt0JpP226e4mnTend7kl/tWA/N5Mw/zILDqxFWTyFqpuKL+JbB222236vPPP2/9dJlxahJYxMGg76WXXlpGP171goDiHX117s0330z/VPHpp5/W3Z7ZaxZYM1t1853x1eiUStK0uFoMgVLChWn2M2/WYdYFVpu4qhNZPF24UhcF1krjiuGbBFb0k58PE1i5/3n5bYE1LzU5Z+WYxs7L4sriahp5WZenSZiDd955ZxLJpDRmWWCViKs6kbVScFcqsJj14r9QEUf8qTz/8MAsVZPAeu+999IM1qOPPrqUdd7nx6zWOeeck2aweHkxv/m8//77S/6Im2vMZMl9+eWXS34/++yzit/MjvE/qfznKv/Nyqb6NodQ5Slp/LO/S7Nrb731Vor7hRdeaAu+4nsWWCuG0BGMA4G6TmM1r1lcWVytJv+6pl3aJulweEKVPX5dPoRRZ1Wa1kr8zarA6iKu+lgWjBivRGC9++67FX+nle/f4l15/HME1/NN7vy3Kdd5CEeO177kceh3FGJr165N/hBycog5+UUInXrqqUu/dZ34m0QS+7722GOPZWF4BY3KwIND43QWWONE13GPjEDXDmWc/i2uLK7Gya9xxN2l4XUVWZMWV5RlFgXWaoorMBtVYH3yySfpfXiIGMTSxRdfnDan88qYKFhKBBazRoimU045JQkdXu/Abz7MIskNE1g8CbnLLrtUZ5xxRgrLS5j5H1XyeOihh6bZLcXFkTal+8yascGeNBFpbMLXPQusiJrPFwaBcXQ6o8RpcWVxNQpvVjtMV0NRKrJWQ1xRllkTWKstrsBMAgthwixP0+f1118foAuvbEG4IKZefvnlgXv4Rexwv0RgKfCwPVjDBBYb9V977TVFl4733ntvygd5yWex+HcKrjMLl+9n4z1evBKC+xZYA5D6x6Ig0HcHxTuqur4M1OLK4qpvHk4qvlHsxDCRtVriirLMksCaBnEFZhJYCIm2D7NLcuxz0iwVLzSuc/xHKfFNUmAhpnLHqyA0E8V+LDlEocr72GOP6fLAkT+0x48F1gAs/rEoCPTZESGu9DLQUpFlcWVx1ScHJx3XqHaiSWStpriiLLMisKZFXIGZBBYihBd5Nn2uuuqqJbqwUV3i5NVXX126Hk+aNrnX7cFSuJXOYLHZvs7x/i7yy5/ayyG2uLb77rsvWzqUH/7dwgJLaPi4cAj02SFFgVXytzYWVxZXffJvNeJaicHIRdZqiyvKMgsCa5rEFZhJYLG8Vur45wgJrKZ3Uk2TwNLSYhRY119/fSoDe7OanAVWEzK+vhAI9N0plYqsRRdXvGkZw9w3/o5vsqJ1pUZCImsaxBVlmXaBNW3iCsxGEVi80gCBxUbwJjftAuuKK65IZTjuuOOailBZYDVC4xuLgMA4OuRhImtaxBXvgzn//PMHPpdcckn6c1j94XMdPow+2VPAX+vU3R92jbh5VJ+/1xnmd9T74/xbmXmMe1Sc+7ARiCw+0+CmWWBNo7iizkYRWHH/UtM7pqZdYPGeLEQis1tNzgKrCRlfXwgERu1YhoVrElnTIq7Iv/77r+69RPwf4dFHH10rgDCohPnWt75VIbaGYZHft8Bavf8cbBKHeR2V/p43IzGtAmtaxRX1P4rAQlRpiZCy1blpF1hsbKcMO++8c/p7oLoyWGDVoeJrC4NAaUcyir9cZG15x1ZLm+D//x6t0f+4eZT85GEksH784x9XF110UfqceeaZFX+I+v3vfz+JqIMOOmiZgEIgMWpjFiuPs+T3JAUWTy6t5qcEj9X0I8E1ah7mzVBMo8CaZnFF/Y8isAh3xBFHJIHC29frHDPkCJguTxHyDirCNO2L0l4q/MnFF402bXJXuLgHi9cy8GoK0uOJx9xt3rw5DVK576cIc3T8eyEQGLVjKQ2Xi6x/PmW4uuKK/Etg8fcZeXnYI7HVVlslkYUhzO+v5LcF1mT3SbXVlQXWoJmbNoE17eIK9EYVWLxhHfHBu6LYkymHMNErGrjfRWDBdcKwt4vXK+ROQqkPgUXcl112WUqPJwl5o7scG/d5kzt54WOBJWR8XCgE2jqfvu7lIotlwr7iXkk8bQKLeJkC/853vpNEFgZQadEJYTCY/tY1HRFPGBZGkMx+nXbaaem/uHSfY5vAYuMocY+6v0vpSDis5uwVaSs/03oUTqPmb96MxbQJrPXr11f6/8C2Y99/f9OlXiWwEEoIjbYPb1iP7qyzzloSIdxjwzhPIyJK9t9//3TsIrDi+7WIh/i6/FVOlxksyvHBBx+k2XzyS/kPPvjgNGuFwOOa3vVlgRVr3ecLg8CoHUvXcBJZ0yKuyP8wgYWfI488Mgks3gOjMvP+F/Zg/eQnP1m6xj1e0rfFFluke3FfF3u1+BsMhW8SWAi2r3zlK9X3vve99Iep8j/KUcLBAqt9tkw4jYIxYebNzaLAWk1xRf1LYCEohn3YrxQdgoj3Y0mIEB6Bxn/46VUOXQQWcTP7JYFDfBs3blxKsu8ZLCJmPxl/rbPTTjstlZ//LnzooYeWniLk/jid/4twnOg67pERGLVjGSXc9ZtuXBIZo4TvO0yJwGIEjVj63e9+t5T3JoH1f//3f8kve7gwcmxS5U9bCf9v//Zv6V/pKUOdwOLpxa9+9atp4zz7HFZaVgkHCywLrC7GYdYE1mqLqy7YtvlFaPH3OPF/A9v8D7uH6HnxxRerd955p/EloMPi6HqfvVw8Hfn+++8vBeW/DBF5LCWO01lgjRNdxz0yAivtyGc5fInAYnSaz1bVCaxNmzal2SeEFHsRIi5bb711ElhaUswFFn+V8fWvf736xje+kUauMeyo5xZY7cJKuAon/e56HLnhTWnAWRJY8yKuppQKxdnK/0dRARF5+j/Fe+65R5fHcrTAGgusjnSlCHTtUObJf4nAYhYKgfWjH/1oSTTVCSxw0ZOHvFurDacosJitQpQxe7XSfVcxTQkHz2C1Cy3hFLHrcr7S9jdt4WdFYFlcTQdzWN7kL4J4JxazcHKffPJJxRPZzF6xt6xuw7389nG0wOoDRcfROwJdOpN581sisPjHewRWfNKwSWDxqgdmoTTjxSZ3wufvypLAwu8PfvCD5B9D1Ce+Eg4WWBZYXYzGtAksHjRhmT7/NL07qktZ7XflCMAX7R/jiL1hWVAzV+wF44nscTsLrHEj7PhHQqDPTn3W4ioRWPwbPIKJZT6Vr0lgcZ+9V2wkjZvdf/jDH1ZXX331UngJLOJlUzub4DmPG+GV1qhHC6x2YSVchZN+dz2O1OimONC0CawphspZ+wcC7BvjKcG99tpraZM7s1o8wfjKK69MBCcLrInA7ES6ItC1Q5kn/yUCixEY4odRmcreJrDkh+Ndd92VnkJk+Y8P+7m4HgXWqaeeWvFqBoQWf53DkmSMY9RzCQfPYLULLeE0Ks5d29u0+7fAmvYamu78vf3229Wrr75aseF9ks4Ca5JoO61iBEbtWOYh3DCBpfsInwcffHBJ+NQJrBtuuKE6/PDDK54GzLHhsWtEGiM67klgxf8i5GWm+PnlL39Z8S6aPI6uvyUcLLAssIqNwQz82XOXstjv4iBggbU4dT1TJe3acc+TfwmouL9K5Tv33HPT0h0b0PGn6xzrBNatt96aBNJ3v/vdiicKo/9tt9023UPscL1OYLFPa8stt0z+6v6eJ8ZXcm6B1S6shKFw0u+ux5lq7AWZ9QxWAUj2MnUIWGBNXZU4QyDQtUOZJ/8SWN/+9rfTS/J4Ud5vfvObit/MJvHqBF6hkJe5TmDhR0KKPVe8aO/YY4+t/vCHP6S4+PPoBx54oFFgEZ5ZMvyxXMim3jzdLr8lHHws+2PpLthGv/NmRSyw5q1GF6M8FliLUc8zV8rYWSzauQQWYkofNpz/7Gc/qw488MDGt6k3CSxmodatW1chsBQfYon/NGTzu/Ctm8HSPQQdYRB5vIFZ17seLazKhJVw6oqv/M9cgx+SYQusIQD59lQiYIE1ldXiTKmj8LFsSakUp4cffjiJKjqs0jD2128dTALPebMgFljzVqOLUR4LrMWo55kr5SQ6Iacxe8LBdVZWZzPX4Idk2AJrCEC+PZUIWGBNZbU4U+5IyzpS42Sc6jhgC2IEjMDqI2CBtfp14BzUIFDXafiaxYQ5UMaBmiblS0bACEwYAQusCQPu5MoQcEda1pEaJ+NUx4GyVmZfRsAIjBMBC6xxouu4R0agrtPwNYsJc6CMAyM3PAc0AkagNwQssHqD0hH1iYA70rKO1DgZpzoO9NkWHZcRMAKjIWCBNRpuDjVmBOo6DV+zmDAHyjgw5ubp6I2AEShAwAKrACR7mTwC7kjLOlLjZJzqODD5FusUjYARyBGwwMoR8e+pQKCu0/A1iwlzoIwDU9GInQkjsOAIWGAtOAGmtfjuSMs6UuNknOo4MK3t2vkyAouEgAXWItX2DJW1rtPwNYsJc6CMAzPU1J1VIzC3CKzRXxD4+ERlDIyBOWAOmAPmgDlgDvTBAQusJ0ykPojkOMwjc8AcMAfMAXPgnxxY4yn3sil342SczAFzwBwwB8wBc6CUA2teeumlyh9jYA6YA+aAOWAOmAPmQH8cWPPKK69U/hgDc8AcMAfMAXPAHDAH+uPAmjfffLPyxxiYA+aAOWAOmAPmgDnQHwfWvPvuu5U/xsAcMAfMAXPAHDAHzIH+OLDmgw8+qPwxBuaAOWAOmAPmgDlgDvTHgTUff/xx5Y8xMAfMAXPAHDAHzAFzoD8OrPnss88qf4yBOWAOmAPmgDlgDpgD/XFgzRdffFH5YwzMAXPAHDAHzAFzwBzojwNrNm/eXPljDMwBc8AcMAfMAXPAHOiPA2u+/PLLyh9jYA6YA+aAOWAOmAPmQH8csMCywLTANgfMAXPAHDAHzIGeObCmsjMCRsAIGAEjYASMgBHoFQELrF7hdGRGwAgYASNgBIyAEagqCyyzwAgYASNgBIyAETACPSNggdUzoI7OCBgBI2AEjIARMAIWWOaAETACRsAIGAEjYAR6RsACq2dAHZ0RMAJGwAgYASNgBCywzAEjYASMgBEwAkbACPSMgAVWz4A6OiNgBIyAETACRsAIWGCZA0bACBgBI2AEjIAR6BkBC6yeAXV0RsAIGAEjYASMgBGwwDIHjIARMAJGwAgYASPQMwIWWD0D6uiMgBEwAkbACBgBIzBVAmvDhg3V2rVrq6233rraaqut/DEG5oA5YA6YA+aAOdArB9AYaA00xzjd1AgsCmpRZVFpDpgD5oA5YA6YA5PiwDhF1tQILNTkpAB1Om685oA5YA6YA+aAOYD2GJebGoHlZUET3cbOHDAHzAFzwByYJAfQHuNyUyOwJgmo03IDNgfMAXPAHDAHzAE4MC5ngeXNg16aNQfMAXPAHDAHFpYDFlgm/8KS36NMjzLNAXPAHDAHxsUBCywLLAssc8AcMAfMAXPAHOiZAxZYPQM6LiXseD3KMgfMAXPAHDAHZocDFlgWWB61mAPmgDlgDpgD5kDPHLDA6hlQjy5mZ3ThunJdmQPmgDlgDoyLAxZYFlgetZgD5oA5YA6YA+ZAzxywwOoZ0HEpYcfrUZY5YA6YA+aAOTA7HLDAssDyqMUcMAfMAXPAHDAHeuaABVbPgHp0MTujC9eV68ocMAfMAXNgXBywwLLA8qjFHDAHzAFzwBzonQP8H98BBxzQe7wrFUQHHnjgRPJkgeVGNXai7bvvvtWVV15ZHXLIIWNPSw1v//33ry677LKU3qWXXjrxRn7sscemMu+4444TK7PKvohH/rkeju29997G2zZtbBxAMMCzk046aWxp1LXfc845p7r44osnmmZdPrrY8m222aZ64IEHqgcffHDV8x3LstNOO1Wffvpp9ac//WlZvrbddttUv7vvvnt1+umnV8cff/wyPzGuYecWWAXGaI899qj+8pe/VI899lj1wQcfVC+99FJ1/fXXV1TUMIB9f6vqj3/8Y+LZmWeeOTG8TjzxxOrDDz9M6VFnMoiHH3549cILLwx8nnvuueree++tLrjggooG1kedXXPNNanMe+65Zy/xrSRPdArnnntu9fzzzydMXnvttermm2+udtlll6K8YWxuu+22hBmYvvLKK9Wtt95a0S5Wkq8+w5599tkJ76OPPrqXPG233XYJs40bN1Z///vfKzCDI/vtt18v8fdZ9mmIq65dxXZ21113jQU3OPjGG29UF1544Vjiz7FFNODuuOOOiaSn9J9++unqzTffnGiaSjseu9jyG264oXrnnXeqvfbaayDfpfYEuxU5xDk2DMGGyI32i3rhPnY35lfniCnuMxDjGnz5/PPPqyOOOGLA/w477JDql8kAxCFlUByjHFNkY/iamz97phIxrijeO++8s7riiiuqZ555pvryyy+rt956a6CSR6mAaQvDdO7f/va3qk8xVNIo+04XgSWDhCg46qijUkNRXqjD22+/PX3uvvvu6v3330/NgLLTyLrUC3HdeOONA2GmSWAxGMCRz0suuaRCNHz22WfJ4GCY2sp62GGHJYHx8ccfJ/5ffvnl1aZNm6rNmzenwca0zBj1KbAw7GCEw5hTZo4Y5Dqj3IbfNN+r4+2o+a1rV2pfHOHdqHG3hdttt90qBkhnnXXWWOLP07bAKhss77PPPslGnHzyyQP10sWeCGvseOQS/TEOu65VEfml3eZ1xm8mSHDqB7iGgKINRP8SWIhC7tPfx/tdz1OiY/iaC4FFpT311FOpIg8++OABoI855pgkssY1MutakX35R+Hj/vznPw+UdyXxy/i2iba+02XGCsNLvmmIWnNXXi666KKB8lHXNCbcTTfdNHBvWNkRH/k0+LQILEb4DAYQRbEcjABxJ5xwwsD16IdzRozMAAo/3ReOzOrq2moe+xRYqrucr4y8mc1ixgQRtprl7SPtOt6OGq/4kLerUeOb1nDqyD2D1b4aoRnvvB672BNhjdDJ46H/xTEg5p78dhFYiC1cFF1s6cBtv/32SWCxqpGn3eV3imwMX3MhsFDbuCYVi/ql847GlopmmQIjTSNkKjKfEcEYXX311YkUZ5xxxpI6zzu7Un+qcDpB9h0xI7N+/fraJUxm5JQmy0QxTcIyUsA9/vjjKY9xiYtyQkZmRGhATLvGsisflJ8ZHeJiZHncccelOPMOS/6HpQt+p5xySlqWQvywNl6XruLjSAMkfc4RWppObusISIeZypdffjnFT73XNbDzzjsvcQIcqUdmNRhVcX7kkUemNNVJg5/wJj8Rb+W3T84oTh1Z0gKzaES4x9Q4rk1Is/8AhxhTfPGIuProo4+W8QxRx5Ikgw9wyKfhFUdpvbIUzwj0nnvuqa666qqK/XXwgWuKq0lgkRfqizbx17/+NYVVmLojvPrkk0+WCVL5pY5xWmrQ9YMOOijlh3TgMyN43eNIPRCWIxygTuCD2gRL07QnrvHRkrbigLfUA0uX2BSwpR2uW7duIB1mgkknF8TUAdcRiW28VXoltkR+Oba1q+gv4kBboQxgRjujHUS/1AX2AzuKPaFNgzvl2HnnnZNfjvzmHmEJw29sDvuFaMMbNmxIda8wMQ1xEMzJi77XCMUAACAASURBVOKJfuK5OnLyxOwJ/Cb/7I+iU45+OYcH4i72tmlPT5tdJp66JULioqyxfVH+NhvdBX/S7WrLCUObwyZed911A3h0tSfCuk5gkc6TTz6Z2iLYyW8XgUUcDJbiIBE+sAWIe9QXPOJ81E/K4Bi+5kJg0ehxGM86gGlgGPrYsOgEmDFAqTNrwFIMnS+dguJQ5/vII48kY44AYMkFJ4OL31J/+MUQ4RB8pI2DPFEgQfy33367YuQKqRiN49hPQxwPPfRQWjZSWAgcDTVLpDjKw7Qt7tFHHx0QOxh/ys+sB7N/pEU+cLFswmJYunSu7PkBR/CkbMRPuhH3GF/beVtHgHF69913l5YWH3744VQvEmfESwPEeHCPDg+MtFzGOZ0X/lR3XKPDbqpj/PbJmbayx3uawWLvTLwezxEKuFxMRD/5OX4pL0uu1BF1T33lIq20XhEEr776aorjxRdfTFwQp+Cy0q8TWLRb/FJfdFDUEwIaDihcfkSg4OBxfo/ftCfafGxXiEkcZaXMcJ90WaZWHOpc2AfCPQZn5AvHPg8GNHA88iRuahafCE+7pTyU5YsvvljiHGkhzHCnnnrqUtpcZ8CFw2a18Ra/JbZE5dKxrV3JD0f5w/5QfsoLHjhEjvzSFu+///50HVuDP/zT+eEkYDnixC91tM8++2yyhdgO9gHhsFvcVxrMVjADgiN+BgtwFd7LT35U/MRFfZEfxYFtwj4oDEKasmGT4YW2ISCK5IfjMLuMn1xgETd4gGPcNzrMRpfiT5qj2HLCiT8MhPmtT1d7IqybBJaW8VnOk9+uAgsRSJ2P0peoXG3HRK4xfM2FwGKUgit9Eoy9GrgoJFhapJE98cQTS0STsaTRMSKlgjDYGEsaqSqs1B8jGMKSXxGFaxCHa4pPxlmbDiElRg2nkRsdJC6f2dAaNp0Jxo+Prqm8GAqMDp2HMKN8CCNc3VMbyltTuoTFQMUlWjXgpplFxVl3lIGpW8rAaOHUoE877bT0mzIrLjpXnIQU16nfpiXCYXXcN2eUz7oj4kEjUvLMyLvOn66JG7vuumurP/nHH50/HNZsATyR0Y+zaKX1ykwPLhpr6gm+twksRDEdK5yHl+SRIyIo72iVf45wGZfPIEU/8Rw8EW5gRVm5R9qIQYSWxLkEFh252jxt5PXXX0/pcV1thjBwPpZPtkCDIdJh/xsDAjBXuy8RWMp/HW9LbYni0FHtChyon/yT+6NsqhfyDl7gKIGidkZ5havKC2DDBBbiQzaN8MSDI1/KCwNcRKoGEOQDUQe3FL/86qiOnPjj7CH2BMcTy/KLMIRvqm/CUs+xnPgtsctRYNGOZGdV78Qje9xmo1VPw/BfiS1HQOKYQRQWHLvaE2Etexzjoo1QB7QxrstvV4GlSZQ4kRDTWel5AmIMX3MhsOgcESmlIGO4aQi5fyltbciTscwbMbMdIgxxlPpjKpNGm2841gwScXEPR2cX84ewI8+IFq43CR1GaZRPxg6/GEMaOjNf/EaU4WTYlI4adVeBxewFLh/xkQc6FUacSqP0qLzwRBiGiA/5pq5psJRHjQ3jRScUp5ARJRjlaNjqOqrSuuubM204sPQpR4cWy1AXDsOGw3jpPrNCzBbFj0QEIgjHKyrkn6NmhSRCu9QrdZxvRCVOOBsFSD6DpQ5a4l/50cydOlVd15HN2LgoBnWv7shSOTN2+RPFMtwaqEhgRWFOfIygceQ3xo/NwPaocxaf4swZ/iVAJQhXKrBKbEnMp87VrlJhsi/aVe6P+tI1jizf4rRaAFdoizmu6rxlOzni8hmsfPBAfeI0sIKzuDgAJR/YLpzqLeaRc3Xk+d5b2hLl1J4gbBR5U9tQPBJBbD/hWqldlsA69NBDE9+YOcuxKbHRqqdh+K/ElsNdBggqs45d7YmwRoDKVtOumPEnfpz6LfntKrAQ3LgovJXfPo4p8jF8zYXAuu+++xI0MnJtgDNixyDm686EQVjgZFybjCXpYVSUTqk/TW9jVOJHU9c0cpYMYh6URn6sE1gIKQQcYiDGzzlCR0bllltuSWloFKq41ajzzk73Odalq86ijvyaFdEoOMbVdq68UFf6YBgpGw03LuUSD+lQdjCkESOA80bcJrDyDjHW8Tg401Z28gLOzJrxBCxlRuw0hRHGsZOQAEkV/Y8vDRzYJ4OrE26aOSKt0nolXRydfp7HYQJLwoW6ipxlKQ6XCxrFL2FUuveCjo72p/A6MkuMY9TOtSaBxTIgTh2uwlNmnGa1mmyBOggtJwrbtiVCpVHH2xJbovDxqHbFMhBP9uUf+ZW/3BawHIXTkjWDAdkVheXIHjzcMIGVCycGTTg9zaj6YFAb+YENwNVxjvTVkefxcw9OgmnML8IOEc4AjoEC7QCnfVOldpm4sT3vvfdeCq82p7RKbXQp/iux5XCIAZzypmNXeyKsKbBsNbaYmWkGvZRFcctvbpt1X8I2HziBI07tR/77OqbIx/A1FwJLRo2NgXWA05EwimC0ogaskVT0L1JjvLmueNs63y7+EDkoekhX90GEqONQw475i+d1QkdlYzmjLn6MEnFwpCHkHazKnxvVYekqz/mMGOGEoWabYlxt58qLRrJtfrnHEhCOEZTO6cRiuLqOSvlrq2Ph2idnYr7azmXYxck6v+eff34qO6PmuvuadZCxRzyyvFI3IKFzgafEU1qvYIdj83Ge/jCBRV7gYh1fudYkoLiOo77zNPmNIafNq4yUiZni3C9tACzIB/fUoWuQJf8rFViaHUQ0E+dKBVaJLVHe47G0XclfbgtygYUQidsllFZfAkscRAzUcYQOWWnGozryOoEl+4cfwsBbHCKOc9qTBJzssPKh3zGteA7fcYgLBoQsscfVBNmSYTa6FH+VZRRbTt7IR8w/513tibCuWyLM45bfJoGlgWEusNTfkbc8zj5+p0obw9dcCCyMAK7J2LKGz4wTJMTgYtDrljM0mtborKTzpXJL/WEk6OTzxhAJoo6jbQMn/kW4OEVOvJSN0X+MMz+X8aMRx3tNjTr6qUtX4cAh+uWcMrNUF41M7qfut+IsFVjEz1IURpjlGISsOlfFP6rAGgdnlCcdmZFjOSAXeqpTDKn85keN7hBS+T1+5wJLfM3rn9E1YgMMCac6GFavYE846jpPf5jAuvbaa1PbHdZx5fEy8wTXm3DJR8K0dzq8fNZWgkp7cvS7b4ElLLVUwsZ6XNyzRhnjJneVuY63JbZE4eNR+RjWruRvmMBizyp1n+MqG7PSGSzZQwnTWJa2c3XkdQKLPCOACK/BAQ/DxPhyQaV8DLPL8J36otxavovtUu15mI0uxV844z/mvyl89MOsLbY5XuO8qz0R1iUCi/gZHDAzX9cnaHkynywR/nk587yP+js1xjF8zYXAogOkc2UpJV/vhuiIK6Z+BT4jWaYw803BTJfiVx2zOqK804vLR8RZ6k8jpdx4M3MlQ8SUPXmIm+1Jg0elcTzKzm/2QOA0la6yMZqk48mX5Gg0EnYia75MqtFpblQVd1O6LKGxB4dlmOiXpSNwzssS/TSdy0AM6whieMqDsWd6vs6wMnUfeUDY0rrrmzMx35xrT0m+5KGZjmik87AYOLBHVOaGCb/aD6MZLKWFEI1xKS09lNClXulYqOvY/uh0MaRte7C0dKYlOuWHJ67y5Tjd05FRMPWt/UC6TliWrcBDT25pT1f+QkWeDMRpxNyXwMrzzrIL7VIzuRqo5C++1QZj1RVlquNtiS0RHvFY2q7kL7cFshEahIpb+fYAbCROdo0jTrPA6pTzdqoZHtk12UNseywH522iXPFjD2XPCYNdhDMSA7KrypfS0DKZ0lA+clum8LLLtAM2zSse9qjRLuLsfomNLsV/JbZcTwvm4rirPRHWwlRlbzpig3HCVv6wHQyCEGC5+NLeTfFJYfo6pgyN4WsuBBYgYzipHAwr/wdFg6cD4ekdRhSx48H4QXo25eGPRqKp1ih+SjvfUn8QCPJgMGmQ5INZNzohZtlEFnUGGH9IyLQoU7mUhU4Pf5CaeBjJ0sgkFvGPIQcHOkx+05DAJk6n83g017hHY6ZT5TeubZN7U7paQuFdNuAJrhgSOjl1KipfyVEGpovAkuikDNGgKT2MHZjBFfGhtO765ozypCNCAD4yoqTzpMOnvqh3MMxFvsLpSMcBj/BL58QGdspJXNQrAizu0ZLhZ2aAzpI6h5vkIRrc0nqVMIGPpMvMDJ0Nrk1gkX9mzGiPdNY88cVToSzXwPco2FRWHTG25JkyM/MjHlNWXJwdokzMWvAIPrNIlBnuk24UdypHtAOkJxzggdLn2LQHi7TAlHSYpaNNgrnC0ukzEICP5J0ZLT1BR96jwKrjbaktUXo6ql3x8Ah2Mv+APX7lb5jAwu7w8AD1QCdIu5c4oRzqEDniJGTUKQ8TWOSF2UUcoo12Td6oM4RS3kmrnIof3OEXS+18eB0E11SP4Ah/qAfaDfUucUiakUMldjkXWNhr2iX4aB9liY0uxZ/yjmrL9YAJWyqEm45d7ImwLhVY9AfYJDCnHuEMdlivFIqYKz/wBNtIWrrW5zERbAxfcyOwABvjTAeB0cRxpHHlI1z5ZYMmjY0PRjlfYiztfEv9kS5LG+pQyCMzP4gSzS7hB/VOY6fh4yAjU8r5rBQdMOFxUVAgcDAkchj7fIqdzpZ8UHYceym09t4msMhfU7oYWAwJjjyzxt9kAImn7SMD00VgER/1SB7yERD3iFONmKl1rnWpO/jVF2fqyk790vGJv9QNxrNpb1UeB6IRg4VBxxGeekUERH4Rjg6e2ROMFg6u0bGoE4hxl9YrvBFn6fzIC+2R/Cg+jUSjUSdv5FF5gTt0VBLBClt3BDOWdxSWMlNHGO3cP34ZPZM3HPliEBONdl8Ci/ar9ylRn3Q+UbiSN9qsNkOTH7DSjFoUWHW8JXyJLckxULtKANR8afZF/oYJLOJnNo4ZJhz4a0M6v/XE9EoEFmlgm2RbSAP7VlfHKq86ffCkTanOiYMBgPxxxEYhYuEdccM92UIJQvyV2OVcYBEOMScbLrs0zEZ3wX9UW44NYBBDO4146LzUngjrUoFF/Ng0rQqIN0wK0P6UfjwySMtn9+P9lZ4n8o7ha64ElkDGYNOgo+HUvfzICEazP/m9cf6G3BgfNbimtJi5yDvH6Jd7+KmLh7LloiyG5Rw/dZ1q7i//3ZYuabblOY+rr99gQEPMl75i/PgB9xJuxHDxfNycYTaLd9PkHXLMw7DzYbxReOFRxx/50bGkXhUf/IYDzG7Q6SiOtmMM2+av7h5hafMlvCNvw2YE69IouZYLdtJpq0eVGU61xS9/dbwttSVt8fdxD36oHAyuEJbkrY+4FUdMQ9dKjuQDQdrml7xrhaDNH/dK29eweEiTMg3zV3KfuLracmZPGQgMq6e+ypuXg3TZfyre5Pf5jZBGIA+rv7qwpdfGoK1SlHMpsEpBtb/2WaRZw4fZLkagcfQ/a2WY1fwyImWEqT1PlEMbhVn6m9Vydc13LrC6hp81/wgStlfEjp0VA8RVqbCetTLPU34R/yzVtQ1KV7O8DCzYapK/z6zvPFlgDVlS6htwxzc74ou9ZoyYmYbPNw27HidTj+wZw1Azg8hSo94jxJJtyazSvNTTogksZjZ4OIitCizVsqSNY6m4bmvGvNTzPJUD+8nAlD1q01Yu3vPFUj4PGYwzbxZYFlhjJdg4yTvuuNmoi2Fnb9gidebjxrVr/HS21AECi32FPHEWZ7S6xjeL/tkkzdON8YGCWSxHlzzT5nhoh83t7MNhf+O4O8Qu+bPf4YMs9kayV22asOLlvezL4h1y486XBZYF1thJNm4SO/7hhs4YGSNzwBxYDQ7U7e9bjXzENCeVJwssCywLLHPAHDAHzAFzwBzomQMWWD0DGlWyzz1iNAfMAXPAHDAHFpMDFlgWWB61mAPmgDlgDpgD5kDPHLDA6hlQj1QWc6Tiene9mwPmgDlgDkQOWGBZYHnUYg6YA+aAOWAOmAM9c8ACq2dAo3r1uUcz5oA5YA6YA+bAYnLAAssCy6MWc8AcMAfMAXPAHOiZAxZYPQPqkcpijlRc7653c8AcMAfMgcgBCywLLI9azAFzwBwwB8wBc6BnDlhg9QxoVK8+92jGHDAHzAFzwBxYTA5YYFlgedRiDpgD5oA5YA6YAz1zwAKrZ0A9UlnMkYrr3fVuDpgD5oA5EDlggWWB5VGLOWAOmAPmgDlgDvTMAQusngGN6tXnHs2YA+aAOWAOmAOLyQELLAssj1rMAXPAHDAHzAFzoGcOWGD1DKhHKos5UnG9u97NAXPAHDAHIgcssCywPGoxB8wBc8AcMAfMgZ45YIHVM6BRvfrcoxlzoBsHzjnnnOriiy+2obddMgfMgZnngAVWC4nXrVtXvfDCCwOfxx9/vLr99turI444orXy77zzzurZZ5+ttt9++1p/F1xwQYr3vvvuq72vjnmPPfZYSv/ggw9u9asw8Xj00UdX5Pnvf/979dprr1V33HFHtdNOOw2NZ5tttkkd3UsvvVR9+OGH1SuvvFJdccUV1Xbbbdca9i9/+UvK7xlnnNHqT3ncfffdq9tuuy2FUTq33nprRbnlh+PWW2+9hIPq5Omnn642bNhQnXbaael+9B/P99577xR27dq1A3Hi58gjj6yoq6b87rnnntU999xTvf766wmH5557rjrrrLNa0zvmmGOW5VV51vHYY49dygtceuyxx6p333031RPnw/gVy0feH3300RT+vffeq5588snq1FNPXYpffsU55eH555+vHn744eq6666rKKf81R1z/MHh5ptvTunssssuy8JSf2+88UZ14YUXLrtXF7+uUadvvvlmpzAKmx+pW3H/7bffrh544IHqwAMPbIx7GBfy+Pk9rO66cqEujW233Ta1Per1gw8+SLhSljo+075Vvxypp/vvv79av359tcMOOzSWnXSvueaa1BbzPBDuvPPOq2iXcDO/P6qtyOPhd5v9gL+UBwwo2+WXX97aDmP8hx12WIWtxQYSHu5fcskly+zz4YcfPoAf6cjmcy/G6fNug6dFxMsCq0VgHXfccQmfZ555JokqhBVGbvPmzen6X//619oGR4f+5ZdfJj9nnnlmrZ+rr756CftDDjmk1g+ERNTIdel0CfvHP/6x+uKLL5Ix+fOf/5wM6Kefflq9+uqrFUaxjfDXX399SpayX3rppdUTTzyRfm/cuLEx3EEHHVR9/vnnyV/JLARGD+H38ccfJ5GDwdy0aVPCFyMIjsoj+cXR+VIPfB588MFkLLl+1113LflVGB2vvPLKig4WkcA1On9wRQDIYbzlX8e99tqr+uijjyowQ8gRBqGFwzjLX34EB0RL3QfRDTcOOOCAFP6kk05Kv8EBzG+66aYk5OBYSX3ToeLoeOAUnSBCDUd8MW/iHB0N+BH2xRdfTOlTxjbxUYf/U089lbB5//33l4XdbbfdUp4QozEPw877ElgnnHBCKhd1TP0jBsW1yKtSLtTlu6TuunChLg3EK2IAziCqaCPU3yeffJKukYcYjvZKG1QbYXDw1ltvJT4wSGoa8CGi4ABCKsaHbUKUYEfgGPyK9zkfxVbkcfC7zX5ga3GIRUQ77R1HOeviiteYFQUTbAfti3bwt7/9LYWnPcZBIzYTF20+tk927eyzzx6aXkzb54stwhKZxvC1ZgxxjhTlSggugXXRRRcNNKpdd901GZ3YUcZ0MAY0SAwTjTPe07k6OwxXndHCH4IAYaDGXdLhKn6OjLyY0YhG9fTTT084Ns3YEA7/dPB0vjE+Zklw++6778B1/JBXDDCdGK5EYNFxIKTyjl1GLo6W1cHTycQ8kS7x4CRa8vt0MOCt61dddVXqTOh8EI+4OoHFfVwcuWKMyTOzbYqv9Ehe4QQ4Ksw777yT4opG/tBDD03pIiDlr+6IiMDBH+KWH87pIHCRM+IcHZn8clQ8beK5CX/ECvWOqNtnn30G4o1plJ73JbDAGeEeuc9sEi4Kz1Iu1OV/JXVXx4W6NCRe8vaKDaJ8DABiGal3+JnHpXjOP//8ZffwiwhBYO24445L90kDIccsds4ZxT+KrVDYeASPJvtBGpQzbw/wHjvVNvvKDDnlIu585p4ZO9xll122VGbZnjqbjx3GvsV8+3yxBdSw+k8EG8PXXAssQGUGA5fPUGEo6NAxBswaIcLqOh51dvjDUNZN36vjQ1TgYmeJoWRprKmCMUqkfcsttwz4YbkBg3P33XcPXI/xIHhwjPzjddLEYYTidc4pKwaIGQvcMIF1/PHHJ395GooXcYVRlVFs6uDxz7ICDrwUXkfymtfBfvvtt4Q3mOLqBNbLL7+cBJHi0pHRM3HGzkj32o7MNuC0PIjxB7O6UTgcYnmiLT46DbgjjKLfo446KtVzFJbiXN5ZwlniaUuvDX9EFuVAkCoPO++8cxK1iBpd48is5Q033FAh5pjhy8V1ncCCK+Rd/Efgwxvwi3HrnPKQBqJB1zhShs8++6x66KGHlq6XciHGw/lK6y7nQh4/v8EQXJk1r7vPbBZtmYGg7jcJLHDHRXGpMBwJl9sEBAjx19kvhR3FVihsPLbZD9mKfNlbbbdtVgmO4dgqEdPjnHbDAJSZWN1rEljcf+SRR1JcdbZa4X204IocSIQZw9fcCywMAu6UU05ZapwAS+ep6xhhRlh1IkKd3cknn5z8n3vuuQPxEBcdASMm7uHUwdCBYPhY/oiVGc8xijjER7zOOctcLMXl1/WbjogZGjo7XeN47733pqWCvEMnLfJDOWXIhwksRo24un0kMU2dt3XwzBKCMyNu+deRjrxpFhE/MtJ1Aktx5Ec6PEb1+fVhv+nEWJoY5o/ZLDrWG2+8sdEvHEAssIdqWHy6L87lAgsxhmMZTX7zYxv++GXpJvJR/IvcB2OEKZyGf+SfeosDhVxgIZrBAsHN4IC0xJ18cJPnOf9NJ4trWt4dhQt5GiV1R5gSLrAHFMf+qTydpt9NAktbDXJ7RTzM/OLggeKV6GYwQbuiHrBV+d7IrrZC8cfjMPuBUMZhW2I48oKjbPF6PIc3DB7itbbzJoGFqCKekvbbFr/vLZYASwQdw9dcCyxGlogUOog4PU/jYa8O4kRLPiwHxf0/amDq7DASLGewzKV7HLlO58MIjH0ROAks7tNJYphimHiuzqJOuJEW+zGi//yczpD02e/B8gQbwenoYoepMCxFsq+LMpcKLGZtcHWiSPHGozp4Zm1Y5uSDiJO4qhOS7F8hz20dsXAqFVj4BxeWFmP+hp0rnZI9SQgAhAhhmuJV59K0vFwXTpwDK/CjXgmP4KHjiHuT8vDCP1+ilT8tt2hWKRdY6rBZduaccNQPy1kstSmeKLDo8GljiIbYzjjnngSXwtYdWT5iiRdRhgBklo79YXV+VUelXKiLo6TulM4wLlA/uChA69KM18CKZT21EcQJbRgcGWzILsUw2BjsWbwmfiFQtN+LvNCecpvSxVbENHQ+zH5IUDPbqDAc4QEun6WPfrCtw2xd9C+BhbAUhiwjEwdc1exzDOPzxRJNXeo7EXQMX3MlsJixYDaBDwaHfVPM2OQdNzM7GLK45CMjScONFaPOjk5Ns2FxNodRmZa22A+Aa+twY9yca1YiN4bcQ2BhePIw8TedIDNo0dH50dFGf0zPRzFQKrC07BnjQzQSX/zoCTV18DE/OqfTpEOI+eKczbD5HpXcjzq7kk6VvCCW6cRivvM4636zFIyQqevgon/yA7/o9OL1/JwnSnEIm3gPTkb84hKdOCfc4rFpb47iFv5NAguxi+NJPMLkAktL02wqlsDCH5uomR3RNQks9qFp/08+Y6o8lRyZlZMjvryTjnF04UIMp/PSuivlgjCNtoMBSaxfzuMyK9ysc7SDuJdQeaZe4SUiRtc4avaMwQTtiHSpUwY4XItpltqKGL/Oyf8w+6HZt3zvpwQWTyErvvzIYBdM4nXwjBhio3VfAqsOQ0TXsPareHy06IID43JzJbBYp2c/Dh+emMKpI4kNiU4ah6HVdYwAU8v5BmJ1dkzP03EjzGQoMFh05Ig54tEeoxiv4m86kj9ck8AaNqqjE2DExj4POkHywEZmZoxkZJitwIDFWZRSgcWMGE4CinJoX1u68Y8vPWEZO3jO+dDxsi8DgUU+8hkYZkuEaRNOpZ0q9UjnDyY8XdgUX9116phOBCzr7usaHQidHSKE8ul63ZGZGVy+rIfwj456VHhxDkyFIaKWWUkc+5bkNz/iH9cksBjl4zSrmgss4tNMBDxisAL2ElZKTxjT5nCqf93vekQYIAYQkOCK0IhLYTG+Ui7EMDovrbtSLhDvn/70p4RBXNZT+0o3/vEVl+MRE3BU9UtbZWmUWSyEUYyLNNgLxvV8gEK7wuVPSiOGcZErJbZCOMVjqf2QwMrFsQRW2wwWs5b5sp4e1kkFqapkO5QvCSwwFYZgA3/AtW2fouLw0eJKHBDH+j7OlcCKT5QwCsRFUSEw9TQb+0uY9taHzp+pdZYW5VednfbDsMEUIcZav4yYjKE6pi4CS/sq6pbOmL1q2jhL/vQUW763QZvcyR/+MNqUixHgiSeemD7sF8Exi8e1ptkHxUVawiQe9Vi2Oti2Dp50cHGvCrOBODqkGG9+XtKpIgIQFojgOmGdx5n/5nUIhI1iMvfDshXLNHQIWmbL/eS/EWNte7AQdXUCS5yL8UmkgnO8rvM2/PHD/jzyI/91Aot74I0o1KskKG/kAAILx7Ih3KIt5SJMaXQ90lEiJvLBjuIp4YL8xmOXuivhguJua8P40Sx1ncBSHDrCPWZGc74wSx2fapV/2SBEnq5xRLAh4hVPqa2Icei81H5o4Bp5Qhy0E1wuAhU/R+qamct4LZ5jB7HPuiaBFW2+7iEqcXGlQfd8tKiq40AizBi+5lZgASIbejHUL+faqAAACTtJREFUGECBSqeFQ2RhOOJHIyam2uU/F1gy7sw4YbzogNTZjSKwNLqLI03SprNCyNEhKi/5UTNmuTihg8Ix+6D404WWrzwOpYVwwjUZxy4CS7M57H9T/HTi+chV9+JRuLctETILRn1LWMbww87BDKHQNpOGqIY3zNpoBmhYvNxHPMUnLfMwXQSWZhTzWQLF2SawmCUiH1HMNQksxQcP6cyYEabcuo7Agp+E19J5E0cUJj/ydCdLQHXcY4mL2eE8DL9LuJCH61J3JVzI46fzbxoMdRFYxIuYxa4oDQQKvM6fzuP+/vvvn9pn/mQhAyaEmuxHia1QevHYxX5I7DGQinFoEMVgLV6P51pmbZq17CKwNKNYtyoQ0/S5xZY40NI1rujWXAsshBWGKT7urdFNXQdFZ8KIPE4v5wKLCkEQMLtE3Cy5qJJGEViEJT2MapwB0JNUbUaJzZ24fElLT/NopojRbP5RJ4Uf7sW0VR6OdNh6S3wdZsKnZAZL+HAkbtKlY6KDjmnWnSu/TQJLe+HykXxdXHXXrr322rQ8GMV49MfeJJaCyW/dzFL0m59rGSdfJsQf+JYKLDo7BBKzbBL1eVpNAguBodm92InlAgveUcYcZwk7RBppIrDim9wRbbSHuJcsz1v+m/pHBOQCmzS43iRYhnEhT6dr3Q3jQh4/v6lbXJ0I0hPLJTNYKpu2HRA3S/II3LqHBahvZlSZlYxtWLaBsMSh38NsRV3ZctvBb+Uz2g/qjUEKXInxsLQNx5vaFn5ZukWwI6zhagzPOXyDl7reNoPFigSOPMq/jxZTbRxIhBnD11wLLADVU3B0KhgjDBGNuAlsjCtOgkECInaqmgqnQ4kvz5OAiA2bdw4N2witDfZMk7O0xWPWjN7ZSxCNDW83jhvzuYdxxR+bqCkjBpTRL+VkJN5UTmYNcNHoN/klHt73hIHDYNNhkEfe1YNBRYBpWU0dPKIRoceHNOgw6DQpl5ZgKTfh1Wk3pc91GfS84+cesyA40qCDyz+qD0bSdNr6rfSYSUG4aDlF1+ORGQIcAj2Pn9/MJET/+bm4gRCh3AgRsAE7eBSFtDiHaBSGzEaCHa5uOVnp5fiDF7xBDIF1PruXCyw6cdKBP3SeYEUcLDdFIZQLLOoUjhBOS6eUEbxj21E+dZQwYaaFvCGQCYPL86owbVyoq+MudVfCBeUjHsGd9y9Rl9QVeee9UNQVmNBG4+Z19mCBqeqXJ5CZ2UVkEAeb1xU/g7l8hlv3OMI/HANJBDJiioEi6eodcF1sRW5nYlo6b7If2CfEFNylvHAH3kXBqDjyI3nHLzyjPWDPaCusRODi+9sksOCNMMT2sZ8T19aW83T92+IrkWYMX3MvsBAHjPhZ2mGvFC4uAeaNSy/kw8hwT51d7CQwWnF/g+JQJ6oOHKOLocufjpH/eMQQYYRxGCgMRT7iozNGUMWRKiM/DDthFJYX8kkgxjTieZOBjH7iObNXGE86UaXDrBsdJDMr8qsOPnn6xxfCinwTPoo+RpoYT4VtO7Z1qoiWNidjS/3TeeWP3Wv5BAPflAcEWJtrEz2KkzpmrxKdCI7OlA3dcUYJv+JcTA9u4JfOU/HVHXP8SQsxRJx1S3G5wCJO6ogOkXaDQ3CxdBofTsgFFuGIn/SoVzhKJ0ndNwklwjAbwiww+2/kSK+tnG1cqKvjLnVXwoU63LlGO2AwRdtVe0TkIHzyBy6wCdHhn4ERXI31pLJG+1OXPq+IIC0cccGV3H6U2oo6O5On2WQ/qE/aubgDH2jjEnp5PPlvhBP+GczhCI/gym22BFby9I8vDQKww9Em5Wn4twVVzoHIoz7P50Jg5WBN0286vCiIhuWNGbE4axX9Ew/xxWs6x7AhgiZhWMjjStOhs6YjaOt8VbY+j+DUZ3yjxAV2ceZzlDgmEQau5cKga7qleMNt0iqZzRyWh9I0h8WzkvvM6PZRFpbbGByW5gVeDRMzw2xFm50pzQfcQbzXLWuWxEEesBFN9q4kDvuxkCrlQJ+iKsZlgdXyJ9KllWN/3RsyIoOZEgyp8euOnzFbDMwQaU1P+JoDi8EB1/P46zmKoj7PLbAssCxwzAFzwBwwB8yBheVAn6IqxmWB5Ua1sI3KI8PxjwyNsTE2B8yBaedAFEV9nltgWWBZYJkD5oA5YA6YAwvLgT5FVYzLAsuNamEb1bSPqpw/j/zNAXPAHBg/B6Io6vPcAssCywLLHDAHzAFzwBxYWA70KapiXBZYblQL26g8Mhz/yNAYG2NzwByYdg5EUdTnuQWWBZYFljlgDpgD5oA5sLAc6FNUxbgssNyoFrZRTfuoyvnzyN8cMAfMgfFzIIqiPs8tsCywLLDMAXPAHDAHzIGF5UCfoirGZYHlRrWwjcojw/GPDI2xMTYHzIFp50AURX2eW2BZYFlgmQPmgDlgDpgDC8uBPkVVjMsCy41qYRvVtI+qnD+P/M0Bc8AcGD8Hoijq83xqBJb/9Hf8JHJDNcbmgDlgDpgD5sA/OYD2GJebGoG1du1az6R4Ns0cMAfMAXPAHDAHJsYBtMe43NQIrA0bNkwMUKv3f6p3Y2EszAFzwBwwBxaVA2iPcbmpEVgUkIKiJr1c6Ma+qI3d5Tb3zQFzwBwYLwfQGGiNcYorNM1UCaxxqUjHawSMgBEwAkbACBiBSSJggTVJtJ2WETACRsAIGAEjsBAIWGAtRDW7kEbACBgBI2AEjMAkEbDAmiTaTssIGAEjYASMgBFYCAQssBaiml1II2AEjIARMAJGYJIIWGBNEm2nZQSMgBEwAkbACCwEAhZYC1HNLqQRMAJGwAgYASMwSQQssCaJttMyAkbACBgBI2AEFgIBC6yFqGYX0ggYASNgBIyAEZgkAhZYk0TbaRkBI2AEjIARMAILgYAF1kJUswtpBIyAETACRsAITBIBC6xJou20jIARMAJGwAgYgYVAwAJrIarZhTQCRsAIGAEjYAQmiYAF1iTRdlpGwAgYASNgBIzAQiBggbUQ1exCGgEjYASMgBEwApNEwAJrkmg7LSNgBIyAETACRmAhELDAWohqdiGNgBEwAkbACBiBSSJggTVJtJ2WETACRsAIGAEjsBAIWGAtRDW7kEbACBgBI2AEjMAkEbDAmiTaTssIGAEjYASMgBFYCAT+H6XBUZltKG45AAAAAElFTkSuQmCC\" \n",
        "width=\"50%\">"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_FgrIBGIPTYT",
        "colab_type": "text"
      },
      "source": [
        "## Checking GPU availability\n",
        "To find out if GPU is available, we have two preferred ways: \n",
        "\n",
        "\n",
        "1.   **PyTorch / Tensorflow APIs** (Framework interface)  \n",
        "  > Every  deep learning framework has an API to check the details of the available GPU devices.\n",
        "\n",
        "\n",
        "2.   **Nvidia SMI** (Command line interface)   \n",
        "  > Nvidia is the manufacturer of the GPUs currently used for Deep Learning.    \n",
        "  Nvidia provides a command line tool for their System Management Interface(`nvidia-smi` for short)\n",
        "  \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dk2AGJUBNQab",
        "colab_type": "text"
      },
      "source": [
        "### pytorch"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t3wQ_2YHMZwQ",
        "colab_type": "code",
        "outputId": "b86e18c8-c2bb-4d31-c9c3-737005b020c9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "import torch\n",
        "torch.cuda.is_available()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "True"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 9
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q_9cheI9b-1w",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "### nvidia-smi\n",
        "If you own GPU you may be familiar with `nvidia-smi`, NVIDIA binary to print out gpu's utilization summary.\n",
        "\n",
        "`nvidia-smi` is now available by default in the colab environment.  \n",
        "Running `nvidia-smi` gives the details of the connected GPU runtime."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "hauEP36SOK6d",
        "colab_type": "code",
        "outputId": "7f1bd3c9-5feb-413c-da6e-90a5f80c3a5f",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 318
        }
      },
      "source": [
        "!nvidia-smi"
      ],
      "execution_count": 2,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Sun Apr 19 17:59:57 2020       \n",
            "+-----------------------------------------------------------------------------+\n",
            "| NVIDIA-SMI 440.64.00    Driver Version: 418.67       CUDA Version: 10.1     |\n",
            "|-------------------------------+----------------------+----------------------+\n",
            "| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |\n",
            "| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |\n",
            "|===============================+======================+======================|\n",
            "|   0  Tesla P100-PCIE...  Off  | 00000000:00:04.0 Off |                    0 |\n",
            "| N/A   32C    P0    26W / 250W |     10MiB / 16280MiB |      0%      Default |\n",
            "+-------------------------------+----------------------+----------------------+\n",
            "                                                                               \n",
            "+-----------------------------------------------------------------------------+\n",
            "| Processes:                                                       GPU Memory |\n",
            "|  GPU       PID   Type   Process name                             Usage      |\n",
            "|=============================================================================|\n",
            "|  No running processes found                                                 |\n",
            "+-----------------------------------------------------------------------------+\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "K5o0gXSmOVHX",
        "colab_type": "text"
      },
      "source": [
        "Violla! I got Tesla P100 GPU w/ 16 GB memory.\n",
        "This could be different for you.  \n",
        "On Google Colab you might get a Tesla K80 GPU with 12 GB memory too."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wKxJCBPsNiDz",
        "colab_type": "text"
      },
      "source": [
        "Now if you want to acquire values in this summary text, youl probably want something else like `gputi`.\n",
        "\n",
        "Next section demonstrates how."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "TJh0jGe2Ncl6",
        "colab_type": "text"
      },
      "source": [
        "## Fetching GPU usage stats in code\n",
        "\n",
        "To find out if GPU is available, we have again multiple ways.  \n",
        "I have two preferred ways based on whether I'm working with a DL framework or writing things from scratch.  \n",
        "Here they are: \n",
        "\n",
        "\n",
        "1.   **PyTorch / Tensorflow APIs** (Framework interface)  \n",
        "  > Every deep learning framework has an API to monitor the stats of the GPU devices.  \n",
        "  It is easier to use this if working with a DL framework.\n",
        "\n",
        "\n",
        "2.   **USing GPUtil python packages** (Custom function)   \n",
        "  > A few python packages like `gputil` provide a interface to fetch GPU usage statistics.  \n",
        "  This can be used if you are not working with any DL framework.\n",
        "  "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "crSV6YsboREk",
        "colab_type": "text"
      },
      "source": [
        "### 1. Pytorch CUDA APIs"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "6S3Achctl7ZH",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 123
        },
        "outputId": "3ea8b2c7-b893-4130-9865-f7c794d654fe"
      },
      "source": [
        "import torch\n",
        "\n",
        "# setting device on GPU if available, else CPU\n",
        "device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
        "print('Using device:', device)\n",
        "print()\n",
        "\n",
        "#Additional Info when using cuda\n",
        "if device.type == 'cuda':\n",
        "    print(torch.cuda.get_device_name(0))\n",
        "    print('Memory Usage:')\n",
        "    print('Allocated:', round(torch.cuda.memory_allocated(0)/1024**3,1), 'GB')\n",
        "    print('Cached:   ', round(torch.cuda.memory_cached(0)/1024**3,1), 'GB')"
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Using device: cuda\n",
            "\n",
            "Tesla P100-PCIE-16GB\n",
            "Memory Usage:\n",
            "Allocated: 0.0 GB\n",
            "Cached:    0.0 GB\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bBjQsif-c1oQ",
        "colab_type": "text"
      },
      "source": [
        "### 2. Using GPUtil python package\n",
        "\n",
        "Let's start by installing a few packages needed to print out our process memory usage..."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "V1wWpNHzGD_g",
        "colab_type": "code",
        "outputId": "1a16de71-5fc2-4e1e-8a45-863040d4e308",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 212
        }
      },
      "source": [
        "!pip install gputil\n",
        "!pip install psutil\n",
        "!pip install humanize"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Collecting gputil\n",
            "  Downloading https://files.pythonhosted.org/packages/ed/0e/5c61eedde9f6c87713e89d794f01e378cfd9565847d4576fa627d758c554/GPUtil-1.4.0.tar.gz\n",
            "Building wheels for collected packages: gputil\n",
            "  Building wheel for gputil (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
            "  Created wheel for gputil: filename=GPUtil-1.4.0-cp36-none-any.whl size=7413 sha256=52a9da8425c2b31a65708cba7214a056bc17e9d99dc55b740d130f0d4ee71c32\n",
            "  Stored in directory: /root/.cache/pip/wheels/3d/77/07/80562de4bb0786e5ea186911a2c831fdd0018bda69beab71fd\n",
            "Successfully built gputil\n",
            "Installing collected packages: gputil\n",
            "Successfully installed gputil-1.4.0\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.6/dist-packages (5.4.8)\n",
            "Requirement already satisfied: humanize in /usr/local/lib/python3.6/dist-packages (0.5.1)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0trsNmJyQt_I",
        "colab_type": "text"
      },
      "source": [
        "Now `gputil`, `psutil`, and `humanize` are all available. \n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "n6LQkC3NbPM3",
        "colab_type": "text"
      },
      "source": [
        "#### `mem_report` helper function\n",
        "\n",
        "Here's a simple function to print out memory usage on **both** CPU and GPU."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vSUyw1sgPtM8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Import packages\n",
        "import os,sys,humanize,psutil,GPUtil\n",
        "\n",
        "# Define function\n",
        "def mem_report():\n",
        "  print(\"CPU RAM Free: \" + humanize.naturalsize( psutil.virtual_memory().available ))\n",
        "  \n",
        "  GPUs = GPUtil.getGPUs()\n",
        "  for i, gpu in enumerate(GPUs):\n",
        "    print('GPU {:d} ... Mem Free: {:.0f}MB / {:.0f}MB | Utilization {:3.0f}%'.format(i, gpu.memoryFree, gpu.memoryTotal, gpu.memoryUtil*100))\n",
        "    \n",
        "# Execute function\n",
        "# mem_report()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H6pP9jJBPjcj",
        "colab_type": "text"
      },
      "source": [
        "#### Trying it out\n",
        "To test the usage of GPU memory using the above function, lets do the following:\n",
        "- Download a pretrained model from the pytorch model library and transfer it to the Cuda GPU.\n",
        "- Then we can run the `mem_report()` helper function to check the used/available GPU statistics."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5TowihFHMCIy",
        "colab_type": "code",
        "outputId": "c3c71179-4a76-462c-87ed-5cf478c007e8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 52
        }
      },
      "source": [
        "import torchvision.models as models\n",
        "\n",
        "wide_resnet50_2 = models.wide_resnet50_2(pretrained=True)\n",
        "if torch.cuda.is_available():\n",
        "  resnet18.cuda()\n",
        "\n",
        "mem_report()"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "CPU RAM Free: 11.5 GB\n",
            "GPU 0 ... Mem Free: 11058MB / 11441MB | Utilization   3%\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "BHswD79baDlg",
        "colab_type": "text"
      },
      "source": [
        "## Closing words\n",
        "Now you can use any of the above methods anywhere you want the GPU Memory Usage from. \n",
        "\n",
        "> I typically use it from while training a Deep Learning model within the training loop. This helps me to get a sense of how much of the GPU memory is available/unused by me. Based on that I can increase/decrease the batch size to utilize the GPU resources efficiently."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "f8acGp54o1Pm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        ""
      ],
      "execution_count": 0,
      "outputs": []
    }
  ]
}