CNN_MNIST.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Convolutional Neural Network\n",
    "\n",
    "A convolutional neural network (CNN) is a a specific type of neural net that can be used to process data with a known, grid-like topology (e.g. 2D grid of pixel values). CNNs got popular due to their excellent performance in image recognition tasks. In this notebook, we will use a basic CNN to classify the MNIST dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import warnings\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from mnist import MNIST\n",
    "import tensorflow as tf\n",
    "\n",
    "warnings.simplefilter('ignore')\n",
    "% matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## MNIST dataset\n",
    "\n",
    "The MNIST dataset contains handwritten digits and is available [here](http://yann.lecun.com/exdb/mnist/). To proceed you first have to download the training data and labels and the test data and labels. Next, unpack them into a folder named \"mnist\" in your home directory. To make the data extraction work, you will have to rename the dots to -, for example t10k-images.idx3-ubyte must be renamed to t10k-images-idx3-ubyte.\n",
    "\n",
    "To read the MNIST files we will use the [python mnist package](https://pypi.org/project/python-mnist/). You can install the package via\n",
    "\n",
    "```bash\n",
    "pip install python-mnist\n",
    "```\n",
    "\n",
    "Note: another convenient way to download and handle the MNIST dataset is explained in the [VAE notebook](basic_variational_autoencoder.ipynb) and involves a tensorflow convience function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_to_data = os.path.expanduser(\"~/mnist/\")\n",
    "mnist = MNIST(path_to_data)\n",
    "\n",
    "x_train, y_train = mnist.load_training()\n",
    "x_test, y_test = mnist.load_testing()\n",
    "\n",
    "x_train = np.array(x_train)\n",
    "y_train = np.array(y_train)\n",
    "x_test = np.array(x_test)\n",
    "y_test = np.array(y_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXAAAAFpCAYAAACFwHNsAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAHwBJREFUeJzt3X+wVXW5x/HPg0qZ5g80kUDFHLRrjWIikZFSoOMlGzGjYlRgdDzOFA025kgOpd7SKH/c6+/kKgLqRW2IJBuvMoqSozEgUQIHxBwl8MwhNQTUdJDn/sHiduL73Z599lp77f3dvF8zzN7nOWvt9azDw8M668f3a+4uAEB6ejU6AQBAbWjgAJAoGjgAJIoGDgCJooEDQKJo4ACQKBo4ACSKBg4AicrVwM3sDDNbY2YvmdmUopICGo3aRgqs1icxzWwPSS9KOk3SeklLJI1z91XFpQeUj9pGKvbMse5QSS+5+8uSZGYPSDpLUsUiNzOe20eh3N3q8LHUNhqumtrOcwqlv6S/dvl6fRYDUkdtIwl5jsBj/zsERyFm1iapLcd2gLJR20hCnga+XtJhXb4eIOm1XRdy9+mSpkv8molkUNtIQp5TKEskDTKzI82st6RvS5pfTFpAQ1HbSELNR+Duvs3MJkl6TNIekma4+8rCMgMahNpGKmq+jbCmjfFrJgpWp7tQeozaRtHqfRcKAKCBaOAAkCgaOAAkigYOAImigQNAomjgAJAoGjgAJIoGDgCJooEDQKJo4ACQKBo4ACSKBg4AiaKBA0Ci8kzoAACFOPHEE4PYpEmTgtj48eOj68+ePTuI3XLLLUFs2bJlNWTXvDgCB4BE0cABIFE0cABIFA0cABKV6yKmmb0iaYukDyRtc/chRSQFNBq1jRTkmhMzK/Ih7v56lcvv1vMG7rHHHkFs//33z/WZsSv1H/vYx6LLHnPMMUHsu9/9bhC7/vrro+uPGzcuiP3jH/8IYtOmTYuuf/XVV0fjedRrTkxquz4GDx4cjT/55JNBbL/99su1rbfeeiuIHXTQQbk+s0zMiQkALSxvA3dJj5vZ82bWVkRCQJOgttH08j7I80V3f83MDpG0wMxWu/uirgtkxc8/AKSG2kbTy3UE7u6vZa8bJc2TNDSyzHR3H8JFIKSE2kYKaj4CN7N9JPVy9y3Z+9Ml/UdhmTXY4YcfHsR69+4dxE4++eTo+sOHDw9iBxxwQBA755xzasiuNuvXrw9iN998cxA7++yzo+tv2bIliP3pT38KYk8//XQN2TWPVq/tsgwdGvyfp7lz50aXjV3Mj91gEatBSXr//feDWOyC5bBhw6Lrxx6xj31ms8lzCqWvpHlmtvNz/sfd/7eQrIDGoraRhJobuLu/LOn4AnMBmgK1jVRwGyEAJIoGDgCJyvUkZo831oRPq/XkybC8T02WZfv27dH4BRdcEMS2bt1a9ed2dHQEsb///e9BbM2aNVV/Zl71ehKzp5qxtusl9qTv5z73uSB23333BbEBAwZEPzO73vAvYr2p0njev/jFL4LYAw88UNV2JGnq1KlB7Gc/+1l02bLwJCYAtDAaOAAkigYOAImigQNAomjgAJCo3X5W+nXr1kXjb7zxRhAr6y6UxYsXR+ObNm0KYl/+8peDWKVHgO+99958iQGS7rzzziAWGyu+HmJ3u0jSvvvuG8RiQzqMGDEiuv5xxx2XK69G4QgcABJFAweARNHAASBRNHAASNRufxHzzTffjMYvu+yyIHbmmWcGsT/+8Y/R9WPjbMcsX748iJ122mnRZd9+++0g9pnPfCaITZ48uaptAx/mxBNPjMa/+tWvBrFKj6jvqtJY8b/97W+DWGxy7ddeey26fuzfYWyYh6985SvR9avNv9lwBA4AiaKBA0CiaOAAkKhuG7iZzTCzjWa2okusj5ktMLO12euB9U0TKB61jdR1Ox64mZ0iaauk2e7+2Sz2C0lvuvs0M5si6UB3v7zbjSU+ZvJ+++0XxCpNshp7Wu3CCy8MYuedd14QmzNnTg3Z7Z7yjAdObf9TbFz82Jj4UvzfQcyjjz4axCo9sXnqqacGsdjTkXfddVd0/b/97W9V5fTBBx9E4++8805VOVUaj7weChkP3N0XSdr1Vo2zJM3K3s+SNKbH2QENRm0jdbWeA+/r7h2SlL0eUlxKQENR20hG3e8DN7M2SW313g5QNmobjVbrEXinmfWTpOx1Y6UF3X26uw9x9yE1bgsoE7WNZNTawOdLmpC9nyDp4WLSARqO2kYyuj2FYmZzJI2QdLCZrZd0paRpkh4yswslrZM0tp5JNovNmzdXvexbb71V1XIXXXRREHvwwQejy1aabR612V1r++ijjw5isaEjKo1///rrrwexjo6OIDZr1qwgtnXr1uhn/u53v6sqVi977713ELv00kuD2LnnnltGOlXrtoG7e6WR2kcWnAtQKmobqeNJTABIFA0cABJFAweARO3244HXy1VXXRXEYuMrxx7XHTVqVPQzH3/88dx5YffxkY98JBqPjbM9evToIFZpmIjx48cHsaVLlwax2IXBlBx++OGNTqFbHIEDQKJo4ACQKBo4ACSKBg4Aiep2PPBCN5b4mMl5HXXUUUEsNr7wpk2bousvXLgwiMUuHt12223R9cv8uy5LnvHAi9SMtT1s2LBo/Jlnnqlq/ZEj488zVZqYOAWVxgOP/dt47rnngtiXvvSlwnOqpJDxwAEAzYkGDgCJooEDQKJo4ACQKJ7ELNFf/vKXIDZx4sQgds8990TXP//886uK7bPPPtH1Z8+eHcRiw4CiNdx4443RuFl4bSx2YTLli5WV9OoVP2ZNdahmjsABIFE0cABIFA0cABJFAweARHXbwM1shpltNLMVXWJXmdkGM1ue/QnHogSaHLWN1FVzF8pMSbdK2vUWhv9093BgYfTIvHnzgtjatWujy8buKog97nzttddG1z/iiCOC2DXXXBPENmzYEF2/Bc1Ui9T2mWeeGcQGDx4cXTb22Pj8+fMLz6kZVbrbJPYzWb58eb3Tya3bI3B3XyTpzRJyAUpFbSN1ec6BTzKzP2e/hh5YWEZA41HbSEKtDfwOSUdJGiypQ9INlRY0szYzW2pm4bB5QPOhtpGMmhq4u3e6+wfuvl3Sf0sa+iHLTnf3Ie4+pNYkgbJQ20hJTY/Sm1k/d9/5DPbZklZ82PLomRUr4j/Ob37zm0Hsa1/7WhCr9Cj+xRdfHMQGDRoUxE477bTuUmxZqdZ2bALh3r17R5fduHFjEHvwwQcLz6lMsQmcYxOLV/Lkk08GsR/+8Id5UipFtw3czOZIGiHpYDNbL+lKSSPMbLAkl/SKpLAzAE2O2kbqum3g7j4uEr67DrkApaK2kTqexASARNHAASBRjAeekNhkx/fee28Qu+uuu6Lr77ln+Nd9yimnBLERI0ZE13/qqac+PEEk4b333gtiqYwLH7tYKUlTp04NYpdddlkQW79+fXT9G24I7xbdunVrD7MrH0fgAJAoGjgAJIoGDgCJooEDQKJo4ACQKO5CaULHHXdcNP6Nb3wjiJ100klBLHa3SSWrVq0KYosWLap6faQnlbG/Y+OZx+4skaRvfetbQezhhx8OYuecc07+xJoIR+AAkCgaOAAkigYOAImigQNAoriIWaJjjjkmiE2aNCmIff3rX4+uf+ihh+ba/gcffBDEYo9QV5r4Fc3LzKqKSdKYMWOC2OTJkwvPqSe+//3vB7Ef/ehHQWz//fePrn///fcHsfHjx+dPrMlxBA4AiaKBA0CiaOAAkKhuG7iZHWZmC82s3cxWmtnkLN7HzBaY2drs9cD6pwsUh9pG6qq5iLlN0qXuvszMPi7peTNbIGmipCfcfZqZTZE0RdLl9Uu1OVW6sDhuXDhbV+yC5cCBA4tOSUuXLo3Gr7nmmiCWylN5ddIyte3uVcWkeM3efPPNQWzGjBnR9d94440gNmzYsCB2/vnnB7Hjjz8++pkDBgwIYuvWrQtijz32WHT922+/PRpvdd0egbt7h7svy95vkdQuqb+ksyTNyhabJSm8tA00MWobqevROXAzGyjpBEmLJfV19w5pxz8ESYcUnRxQFmobKar6PnAz21fSXEmXuPvmSveYRtZrk9RWW3pA/VHbSFVVR+Bmtpd2FPj97v7rLNxpZv2y7/eTtDG2rrtPd/ch7j6kiISBIlHbSFk1d6GYpLsltbv7jV2+NV/ShOz9BEnh2I1AE6O2kTqrdKX6/xcwGy7p95JekLTzGesrtONc4UOSDpe0TtJYd3+zm8/68I01kb59+waxY489Nojdeuut0fU//elPF57T4sWLg9h1110XxGLjIEut+Yi8u1d3viOilWp77NixQWzOnDm5PrOzszMa37x5cxAbNGhQrm0999xzQWzhwoVB7Mc//nGu7aSkmtru9hy4uz8jqdIHjexpUkCzoLaROp7EBIBE0cABIFE0cABIVLcXMQvdWIMv9PTp0yeI3XnnndFlYxOqfupTnyo8p2effTaI3XDDDdFlY48Rv/vuu4XnlJI8FzGL1Ojajj2K/qtf/Sq6bGwi7JhK98NX2zNij9w/8MAD0WUbPR55M6qmtjkCB4BE0cABIFE0cABIFA0cABKV/EXMz3/+89H4ZZddFsSGDh0axPr37190SpKkd955J4jFxly+9tprg9jbb79dl5xaERcxK+vXr180fvHFFwexqVOnBrGeXMS86aabgtgdd9wRxF566aXoZyLERUwAaGE0cABIFA0cABJFAweARNHAASBRyd+FMm3atGg8dhdKT6xatSqIPfLII0Fs27Zt0fVjj8Nv2rQpV04IcRcKWhV3oQBAC6OBA0CiaOAAkKhqJjU+zMwWmlm7ma00s8lZ/Coz22Bmy7M/o+ufLlAcahupq2ZS436S+rn7MjP7uKTnJY2R9E1JW939+qo3xoUeFCznpMbUNppWUZMad0jqyN5vMbN2SfUZQAQoEbWN1PXoHLiZDZR0gqTFWWiSmf3ZzGaY2YEF5waUhtpGiqpu4Ga2r6S5ki5x982S7pB0lKTB2nEUE50HzMzazGypmS0tIF+gcNQ2UlXVgzxmtpekRyQ95u43Rr4/UNIj7v7Zbj6H84QoVN4HeahtNKtCHuSxHYMC3y2pvWuBZxeAdjpb0opakgQahdpG6qq5C2W4pN9LekHS9ix8haRx2vErpkt6RdLF2UWhD/ssjlJQqJx3oVDbaFrV1HbyY6Fg98ZYKGhVjIUCAC2MBg4AiaKBA0CiaOAAkCgaOAAkigYOAImigQNAomjgAJCoboeTLdjrkl7N3h+cfd1KWm2fmn1/jmh0Al3srO1m/5nVgn0qX1W1XeqTmP+yYbOl7j6kIRuvk1bbp1bbnzK04s+MfWpenEIBgETRwAEgUY1s4NMbuO16abV9arX9KUMr/szYpybVsHPgAIB8OIUCAIkqvYGb2RlmtsbMXjKzKWVvvwjZRLcbzWxFl1gfM1tgZmuz16QmwjWzw8xsoZm1m9lKM5ucxZPerzJR282plWu71AZuZntIuk3Sv0s6VtI4Mzu2zBwKMlPSGbvEpkh6wt0HSXoi+zol2yRd6u7/JmmYpO9mfzep71cpqO2m1rK1XfYR+FBJL7n7y+7+vqQHJJ1Vcg65ufsiSW/uEj5L0qzs/SxJY0pNKid373D3Zdn7LZLaJfVX4vtVImq7SbVybZfdwPtL+muXr9dnsVbQd+e8idnrIQ3Op2bZTOwnSFqsFtqvOqO2E9BqtV12A4/N8cZtME3EzPaVNFfSJe6+udH5JITabnKtWNtlN/D1kg7r8vUASa+VnEO9dJpZP0nKXjc2OJ8eM7O9tKPA73f3X2fh5PerJNR2E2vV2i67gS+RNMjMjjSz3pK+LWl+yTnUy3xJE7L3EyQ93MBceszMTNLdktrd/cYu30p6v0pEbTepVq7t0h/kMbPRkv5L0h6SZrj7NaUmUAAzmyNphHaMaNYp6UpJv5H0kKTDJa2TNNbdd70Y1LTMbLik30t6QdL2LHyFdpwrTHa/ykRtN6dWrm2exASARPEkJgAkigYOAImigQNAomjgAJAoGjgAJIoGDgCJooEDQKJo4ACQKBo4ACSKBg4AiaKBA0CiaOAAkKhcDbwVJnEFYqhtpKDm0QizSVxflHSadgxmv0TSOHdfVVx6QPmobaRizxzr/v8krpJkZjsnca1Y5GbG2LUolLvHpjLLi9pGw1VT23lOobTyJK7YvVHbSEKeI/CqJnE1szZJbTm2A5SN2kYS8jTwqiZxdffpkqZL/JqJZFDbSEKeUyitPIkrdm/UNpJQ8xG4u28zs0mSHtM/J3FdWVhmQINQ20hFqZMa82smilanu1B6jNpG0ep9FwoAoIFo4ACQKBo4ACSKBg4AiaKBA0CiaOAAkCgaOAAkigYOAImigQNAomjgAJAoGjgAJIoGDgCJooEDQKJo4ACQKBo4ACSKBg4AiaKBA0CiaOAAkKg8s9LLzF6RtEXSB5K2ufuQIpICGo3aRgpyNfDMl9399QI+B01i5MiR0fj9998fxE499dQgtmbNmsJzahBqOxFTp04NYldffXUQ69UrftJhxIgRQezpp5/OnVe9cQoFABKVt4G7pMfN7HkzaysiIaBJUNtoenlPoXzR3V8zs0MkLTCz1e6+qOsCWfHzDwCpobbR9HIdgbv7a9nrRknzJA2NLDPd3YdwEQgpobaRgpqPwM1sH0m93H1L9v50Sf9RWGZVOuWUU6Lxgw46KIjNmzev3um0hJNOOikaX7JkScmZNEaz1DZCEydOjMYvv/zyILZ9+/aqP9fda02pofKcQukraZ6Z7fyc/3H3/y0kK6CxqG0koeYG7u4vSzq+wFyApkBtIxXcRggAiaKBA0CiingSs6FiT1BJ0qBBg4IYFzFDsSfTjjzyyOiyRxxxRBDLzhMDpYjVoCR99KMfLTmT5sAROAAkigYOAImigQNAomjgAJAoGjgAJCr5u1DGjx8fjT/33HMlZ5Kmfv36BbGLLroouux9990XxFavXl14ToAkjRo1Koh973vfq3r9WG2eeeaZ0WU7OzurT6yJcAQOAImigQNAomjgAJAoGjgAJCr5i5iVJilFde66666ql127dm0dM8HubPjw4UHsnnvuCWL7779/1Z953XXXBbFXX321Z4k1ObofACSKBg4AiaKBA0Cium3gZjbDzDaa2YousT5mtsDM1mavB9Y3TaB41DZSV81FzJmSbpU0u0tsiqQn3H2amU3Jvg5nFS3YcccdF8T69u1b7822tJ5cFFqwYEEdM2mImWqS2t7dTZgwIYh98pOfrHr9p556KojNnj07XLDFdHsE7u6LJL25S/gsSbOy97MkjSk4L6DuqG2krtZz4H3dvUOSstdDiksJaChqG8mo+33gZtYmqa3e2wHKRm2j0Wo9Au80s36SlL1urLSgu0939yHuPqTGbQFloraRjFob+HxJO686TJD0cDHpAA1HbSMZ3Z5CMbM5kkZIOtjM1ku6UtI0SQ+Z2YWS1kkaW88kdxo9enQQ23vvvcvYdEuI3bFTaQb6mA0bNhSZTsM1U23vLg4++OBo/IILLghi27dvD2KbNm2Krv/Tn/40X2KJ6raBu/u4Ct8aWXAuQKmobaSOJzEBIFE0cABIFA0cABKV1HjgxxxzTNXLrly5so6ZpOn6668PYrELmy+++GJ0/S1bthSeE1rXwIEDg9jcuXNzfeYtt9wSjS9cuDDX56aKI3AASBQNHAASRQMHgETRwAEgUUldxOyJJUuWNDqFwu23335B7Iwzzghi5513XnT9008/vart/OQnP4nGKz0FB8TEajM2pn8lTzzxRBC76aabcuXUajgCB4BE0cABIFE0cABIFA0cABLVshcx+/TpU/hnHn/88UHMzKLLjho1KogNGDAgiPXu3TuInXvuudHP7NUr/P/23XffDWKLFy+Orv/ee+8FsT33DEvg+eefj64PVDJmTDh16LRp06pe/5lnnglisYmO33rrrZ4l1uI4AgeARNHAASBRNHAASBQNHAAS1W0DN7MZZrbRzFZ0iV1lZhvMbHn2J5ysEmhy1DZSV81dKDMl3Spp9i7x/3T3cIDpOordceHu0WV/+ctfBrErrrgi1/ZjjwFXugtl27ZtQeydd94JYqtWrQpiM2bMiH7m0qVLg9jTTz8dxDo7O6Prr1+/PojFJoVevXp1dP0WNFNNUtspqcc43y+//HIQq1TH+Kduj8DdfZGkN0vIBSgVtY3U5TkHPsnM/pz9GnpgYRkBjUdtIwm1NvA7JB0labCkDkk3VFrQzNrMbKmZhb//A82H2kYyamrg7t7p7h+4+3ZJ/y1p6IcsO93dh7j7kFqTBMpCbSMlNT1Kb2b93L0j+/JsSSs+bPmifOc73wlir776anTZk08+ufDtr1u3Loj95je/iS7b3t4exP7whz8UnlNMW1tbNP6JT3wiiMUuHu3OGlXbKbn88suD2Pbt23N9Zk8eu8c/ddvAzWyOpBGSDjaz9ZKulDTCzAZLckmvSLq4jjkCdUFtI3XdNnB3HxcJ312HXIBSUdtIHU9iAkCiaOAAkKjkxwP/+c9/3ugUms7IkSOrXjbvE3RoXYMHD47Gq50cO+bhhx+OxtesWVPzZ+7OOAIHgETRwAEgUTRwAEgUDRwAEkUDB4BEJX8XCvKZN29eo1NAk3r88cej8QMPrG6AxtjQERMnTsyTEnbBETgAJIoGDgCJooEDQKJo4ACQKC5iAog66KCDovFqx/6+/fbbg9jWrVtz5YR/xRE4ACSKBg4AiaKBA0Cium3gZnaYmS00s3YzW2lmk7N4HzNbYGZrs9fq7u4HmgS1jdRVcxFzm6RL3X2ZmX1c0vNmtkDSRElPuPs0M5siaYqkcLZTNA0zC2JHH310ECtr8uUmQG1n7rnnniDWq1e+X9CfffbZXOuje93+Dbl7h7svy95vkdQuqb+ksyTNyhabJWlMvZIE6oHaRup69F+smQ2UdIKkxZL6unuHtOMfgqRDik4OKAu1jRRVfR+4me0raa6kS9x9c+zX8QrrtUlqqy09oP6obaSqqiNwM9tLOwr8fnf/dRbuNLN+2ff7SdoYW9fdp7v7EHcfUkTCQJGobaSsmrtQTNLdktrd/cYu35ovaUL2foKk+GylQJOitpG6ak6hfFHS+ZJeMLPlWewKSdMkPWRmF0paJ2lsfVJEUdw9iOW90yBxu2Vtx2abHzVqVBCr9Mj8+++/H8Ruu+22INbZ2VlDduiJbhu4uz8jqdJJwZHFpgOUh9pG6nbrwy8ASBkNHAASRQMHgEQxHvhu7gtf+EIQmzlzZvmJoDQHHHBAEDv00EOrXn/Dhg1B7Ac/+EGunFAbjsABIFE0cABIFA0cABJFAweARHERczdS7SBNANLAETgAJIoGDgCJooEDQKJo4ACQKBo4ACSKu1Ba0KOPPhqNjx3bUsNao0arV68OYrEZ5IcPH15GOsiBI3AASBQNHAASRQMHgERVM6nxYWa20MzazWylmU3O4leZ2QYzW579GV3/dIHiUNtIncUmuv2XBcz6Sern7svM7OOSnpc0RtI3JW119+ur3pjZh28M6CF3r3l8AGobzaya2q5mUuMOSR3Z+y1m1i6pf/70gMaitpG6Hp0DN7OBkk6QtDgLTTKzP5vZDDM7sODcgNJQ20hR1Q3czPaVNFfSJe6+WdIdko6SNFg7jmJuqLBem5ktNbOlBeQLFI7aRqq6PQcuSWa2l6RHJD3m7jdGvj9Q0iPu/tluPofzhChUnnPgErWN5lVNbVdzF4pJultSe9cCzy4A7XS2pBW1JAk0CrWN1FVzF8pwSb+X9IKk7Vn4CknjtONXTJf0iqSLs4tCH/ZZHKWgUDnvQqG20bSqqe2qTqEUhSJH0fKeQikKtY2iFXIKBQDQnGjgAJAoGjgAJIoGDgCJooEDQKJo4ACQKBo4ACSKBg4AiSp7UuPXJb2avT84+7qVtNo+Nfv+HNHoBLrYWdvN/jOrBftUvqpqu9QnMf9lw2ZL3X1IQzZeJ622T622P2VoxZ8Z+9S8OIUCAImigQNAohrZwKc3cNv10mr71Gr7U4ZW/JmxT02qYefAAQD5cAoFABJVegM3szPMbI2ZvWRmU8refhGyiW43mtmKLrE+ZrbAzNZmr0lNhGtmh5nZQjNrN7OVZjY5iye9X2WitptTK9d2qQ3czPaQdJukf5d0rKRxZnZsmTkUZKakM3aJTZH0hLsPkvRE9nVKtkm61N3/TdIwSd/N/m5S369SUNtNrWVru+wj8KGSXnL3l939fUkPSDqr5Bxyc/dFkt7cJXyWpFnZ+1mSxpSaVE7u3uHuy7L3WyS1S+qvxPerRNR2k2rl2i67gfeX9NcuX6/PYq2g7855E7PXQxqcT82ymdhPkLRYLbRfdUZtJ6DVarvsBh6b443bYJqIme0raa6kS9x9c6PzSQi13eRasbbLbuDrJR3W5esBkl4rOYd66TSzfpKUvW5scD49ZmZ7aUeB3+/uv87Cye9XSajtJtaqtV12A18iaZCZHWlmvSV9W9L8knOol/mSJmTvJ0h6uIG59JiZmaS7JbW7+41dvpX0fpWI2m5SrVzbpT/IY2ajJf2XpD0kzXD3a0pNoABmNkfSCO0Y0axT0pWSfiPpIUmHS1onaay773oxqGmZ2XBJv5f0gqTtWfgK7ThXmOx+lYnabk6tXNs8iQkAieJJTABIFA0cABJFAweARNHAASBRNHAASBQNHAASRQMHgETRwAEgUf8HKK758FDCOBsAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x432 with 4 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "fig = plt.figure(figsize=(6,6))\n",
    "\n",
    "for i in range(4):\n",
    "    ax = fig.add_subplot(2, 2, i + 1)\n",
    "    ax.imshow(np.reshape(x_train[i], (28, 28)), cmap='gray')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Creating one-hot encoded labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training set shape: (60000, 784)\n",
      "Training labels shape: (60000, 10)\n",
      "Test set shape: (10000, 784)\n",
      "Test set shape: (10000, 10)\n"
     ]
    }
   ],
   "source": [
    "n_train_samples = x_train.shape[0]\n",
    "n_test_samples = x_test.shape[0]\n",
    "n_classes = 10\n",
    "\n",
    "y_train_one_hot = np.zeros((n_train_samples, n_classes))\n",
    "y_train_one_hot[np.arange(n_train_samples), y_train] = 1\n",
    "\n",
    "y_test_one_hot = np.zeros((n_test_samples, n_classes))\n",
    "y_test_one_hot[np.arange(n_test_samples), y_test] = 1\n",
    "\n",
    "print(f\"Training set shape: {x_train.shape}\")\n",
    "print(f\"Training labels shape: {y_train_one_hot.shape}\")\n",
    "\n",
    "print(f\"Test set shape: {x_test.shape}\")\n",
    "print(f\"Test set shape: {y_test_one_hot.shape}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CNN layers\n",
    "\n",
    "A convolutional neural network consists of a stack of layers. The most important layer types are: convolutional layer, pooling layer and fully connected layer. Typically, convolutional and pooling layers are used in an alternating fashion throughout the CNN. The final, fully connected layer is used to classify the extracted feature values into distinct classes.\n",
    "\n",
    "### Convolutional layer:\n",
    "\n",
    "A convolutional layer is a specific type of neural network layer that performs *convolutions* on an input image. The convolutions are performed by applying a certain number of convolutional filters (also called kernels) to the image. These filters are able to extract features from the input image (e.g. edges or lines). For each input region the kernel is applied to, a single value in the output feature map is produced. \n",
    "\n",
    "Detailed information about the functioning of convolutional layers and several illustrations can be found in the lecture notes of the [Stanford class on CNN's](http://cs231n.github.io/convolutional-networks/).\n",
    "\n",
    "\n",
    "**Output shape of a convolutional layer:**   \n",
    "To compute the output shape of a conv layer you need to know how a convolutional layer is structured. In summary (see Stanford class for details):\n",
    "\n",
    "A convolutional layer:\n",
    "- Accepts an input tensor of shape $W_1 \\times H_1 \\times D_1$ (width, height, depth)\n",
    "    - The depth describes the number of color channels. For example, for MNIST we have only 1 color channel (grayscale). Most images will be in RGB mode with 3 color channels (red, blue, green)\n",
    "     \n",
    "- Requires four hyperparameters:\n",
    "    - the number of filters $K$\n",
    "    - the shape / spacial extend of the filters $F$\n",
    "    - the stride $S$\n",
    "    - the amount of zero padding $P$   \n",
    "    common settings for padding are 'same' and 'valid'. 'same' means that the input will be padded with enough zeros to ensure that the output will have the same widht and height as the input. 'valid' means that no padding is used\n",
    "     \n",
    "- Produces an output tensor of shape  $W_2 \\times H_2 \\times D_2$\n",
    "    These shapes can be computed as follows:   \n",
    "    $W_2 = \\frac{W_1 - F + 2P}{S} + 1$   \n",
    "       \n",
    "    $H_2 = \\frac{H_1 - F + 2P}{S} + 1$\n",
    "   \n",
    "    $D_2 = K$\n",
    "\n",
    "\n",
    "**Example:**\n",
    "\n",
    "For MNIST, the input has shape $28 \\times 28 \\times 1$. In the first convolutional layer we will use $32$ filters with a kernel size of $5 \n",
    "\\times 5$. We will use \"same\" padding and a stride of $1$.\n",
    "\n",
    "Therefore, the output of the first convolutional layer will have the following shape: \n",
    "\n",
    "$W_2 = \\frac{28 - 5 + 2 * 0}{1} + 1 = 24$   \n",
    "       \n",
    "$H_2 = \\frac{28 - 5 + 2 * 0}{1} + 1 = 24$\n",
    "   \n",
    "$D_2 = K = 1$\n",
    "   \n",
    "      \n",
    "      "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pooling layer:\n",
    "\n",
    "Pooling layers are used to reduce the dimensionality of the feature maps produced by convolutional layers. A pooling function replaces the output of a convolutional layer at a certain location with a summary statistic of the nearby outputs. For example, *max pooling* computes the *maximum* output value within a rectangular neighborhood.\n",
    "\n",
    "**Example**:\n",
    "\n",
    "![title](figures/max_pool.png)\n",
    "\n",
    "\n",
    "### Fully connected layer:\n",
    "\n",
    "The fully connected layers are used to classify the features extracted by the convolutional and pooling layers. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CNN network architecture"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Network parameters\n",
    "height = width = 28\n",
    "n_channels = 1\n",
    "batch_size = 100\n",
    "lr = 0.001\n",
    "n_epochs = 5\n",
    "\n",
    "n_train_batches = n_train_samples // batch_size\n",
    "n_test_batches = n_test_samples // batch_size"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Placeholders for input batches and targets\n",
    "X_placeholder = tf.placeholder(tf.float32, [None, height * width])\n",
    "y_placeholder = tf.placeholder(tf.float32, [None, n_classes])\n",
    "training = tf.placeholder(tf.bool)\n",
    "\n",
    "input_layer = tf.reshape(X_placeholder, shape=[-1, height, width, n_channels])\n",
    "\n",
    "# First convolutional layer\n",
    "# output shape: (batch_size, 28, 28, 32)\n",
    "conv1 = tf.layers.conv2d(\n",
    "            inputs=input_layer,\n",
    "            filters=32,\n",
    "            kernel_size=[5, 5],\n",
    "            padding=\"same\",\n",
    "            activation=tf.nn.relu)\n",
    "\n",
    "# First pooling layer\n",
    "# output shape: (batch_size, 14, 14, 32)\n",
    "pool1 = tf.layers.max_pooling2d(\n",
    "            inputs=conv1,\n",
    "            pool_size=[2, 2],\n",
    "            strides=2)\n",
    "\n",
    "# Second convolutional layer\n",
    "# output shape: (batch_size, 14, 14, 64)    \n",
    "conv2 = tf.layers.conv2d(\n",
    "            inputs=pool1,\n",
    "            filters=64,\n",
    "            kernel_size=[5, 5],\n",
    "            padding=\"same\",\n",
    "            activation=tf.nn.relu)\n",
    "\n",
    "# Second pooling layer \n",
    "# output shape: (batch_size, 7, 7, 64)    \n",
    "pool2 = tf.layers.max_pooling2d(\n",
    "            inputs=conv2,\n",
    "            pool_size=[2, 2],\n",
    "            strides=2)\n",
    "\n",
    "# We need to reshape the output of pool2 to fit the fully connected layer\n",
    "pool2_flat = tf.reshape(pool2, [-1, 7 * 7 * 64])\n",
    "\n",
    "# Fully connected layer    \n",
    "fully_connected = tf.layers.dense(\n",
    "                    inputs=pool2_flat,\n",
    "                    units=1024,\n",
    "                    activation=tf.nn.relu)\n",
    "\n",
    "# Dropout layer for regularization\n",
    "dropout = tf.layers.dropout(\n",
    "            inputs=fully_connected,\n",
    "            rate=0.4,\n",
    "            training=training)\n",
    "\n",
    "# Logits (unnormalized probabilities)\n",
    "logits = tf.layers.dense(\n",
    "            inputs=dropout,\n",
    "            units=n_classes)\n",
    "\n",
    "# Loss function\n",
    "loss = tf.reduce_mean(\n",
    "            tf.nn.softmax_cross_entropy_with_logits_v2(\n",
    "                            logits=logits,\n",
    "                            labels=y_placeholder))\n",
    "\n",
    "# Optimizer and train step\n",
    "optimizer = tf.train.AdamOptimizer(learning_rate=lr)\n",
    "train_step = optimizer.minimize(loss)\n",
    "\n",
    "# Accuracy\n",
    "correct_prediction = tf.equal(tf.argmax(logits,1), tf.argmax(y_placeholder,1))\n",
    "accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Training and Testing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Epoch: 0\n",
      "Minibatch loss: 80.091   Accuracy: 0.230\n",
      "Minibatch loss: 0.200   Accuracy: 0.950\n",
      "Minibatch loss: 0.163   Accuracy: 0.930\n",
      "Minibatch loss: 0.219   Accuracy: 0.940\n",
      "Minibatch loss: 0.083   Accuracy: 0.970\n",
      "Minibatch loss: 0.051   Accuracy: 0.990\n",
      "\n",
      "Epoch: 1\n",
      "Minibatch loss: 0.074   Accuracy: 0.990\n",
      "Minibatch loss: 0.017   Accuracy: 1.000\n",
      "Minibatch loss: 0.125   Accuracy: 0.960\n",
      "Minibatch loss: 0.127   Accuracy: 0.970\n",
      "Minibatch loss: 0.029   Accuracy: 1.000\n",
      "Minibatch loss: 0.033   Accuracy: 0.990\n",
      "\n",
      "Epoch: 2\n",
      "Minibatch loss: 0.017   Accuracy: 1.000\n",
      "Minibatch loss: 0.011   Accuracy: 1.000\n",
      "Minibatch loss: 0.081   Accuracy: 0.970\n",
      "Minibatch loss: 0.027   Accuracy: 1.000\n",
      "Minibatch loss: 0.068   Accuracy: 0.980\n",
      "Minibatch loss: 0.011   Accuracy: 0.990\n",
      "\n",
      "Epoch: 3\n",
      "Minibatch loss: 0.031   Accuracy: 0.980\n",
      "Minibatch loss: 0.034   Accuracy: 0.980\n",
      "Minibatch loss: 0.048   Accuracy: 0.980\n",
      "Minibatch loss: 0.066   Accuracy: 0.980\n",
      "Minibatch loss: 0.006   Accuracy: 1.000\n",
      "Minibatch loss: 0.011   Accuracy: 1.000\n",
      "\n",
      "Epoch: 4\n",
      "Minibatch loss: 0.008   Accuracy: 1.000\n",
      "Minibatch loss: 0.015   Accuracy: 1.000\n",
      "Minibatch loss: 0.022   Accuracy: 1.000\n",
      "Minibatch loss: 0.023   Accuracy: 0.990\n",
      "Minibatch loss: 0.019   Accuracy: 0.990\n",
      "Minibatch loss: 0.006   Accuracy: 1.000\n",
      "\n",
      "Optimization done! Let's calculate the test error\n",
      "\n",
      "Average loss on test set: 0.007\n",
      "Accuracy on test set: 1.000\n"
     ]
    }
   ],
   "source": [
    "with tf.Session() as sess:\n",
    "    \n",
    "    # We first have to initialize all variables\n",
    "    init = tf.global_variables_initializer()\n",
    "    sess.run(init)\n",
    "        \n",
    "    # TRAINING\n",
    "    for epoch in range(n_epochs):\n",
    "        \n",
    "        print()\n",
    "        print(f\"Epoch: {epoch}\")\n",
    "        \n",
    "        for i in range(n_train_batches):           \n",
    "            start = i * batch_size\n",
    "            end = (i + 1) * batch_size\n",
    "\n",
    "            x_batch = x_train[start:end]        \n",
    "            y_batch = y_train_one_hot[start:end]\n",
    "            \n",
    "            \n",
    "            _train_step, _loss = sess.run([train_step, loss],\n",
    "                                        feed_dict=\n",
    "                                        {X_placeholder:x_batch,\n",
    "                                         y_placeholder:y_batch,\n",
    "                                         training: True\n",
    "                                        })\n",
    "    \n",
    "            \n",
    "            if i % 100 == 0:\n",
    "                _loss, _accuracy = sess.run([loss, accuracy],\n",
    "                                 feed_dict={\n",
    "                                     X_placeholder:x_batch,\n",
    "                                     y_placeholder:y_batch,\n",
    "                                     training: True\n",
    "                                 })\n",
    "                \n",
    "                print(f\"Minibatch loss: {_loss:.3f}   \" \n",
    "                      f\"Accuracy: {_accuracy:.3f}\")\n",
    "          \n",
    "    print()\n",
    "    print(f\"Optimization done! Let's calculate the test error\")\n",
    "      \n",
    "    # TESTING\n",
    "    losses = []\n",
    "    acc = []\n",
    "\n",
    "    for i in range(n_test_batches):\n",
    "\n",
    "        start = i * batch_size\n",
    "        end = (i + 1) * batch_size\n",
    "\n",
    "        x_test_batch = x_test[start:end]        \n",
    "        y_test_batch = y_test_one_hot[start:end]\n",
    "\n",
    "\n",
    "        test_loss, test_accuracy = sess.run(\n",
    "                                        [loss, accuracy],\n",
    "                                        feed_dict={\n",
    "                                                X_placeholder:x_test_batch,\n",
    "                                                y_placeholder:y_test_batch,\n",
    "                                                training: False\n",
    "                                                })\n",
    "\n",
    "        losses.append(test_loss)\n",
    "        acc.append(test_accuracy)\n",
    "\n",
    "    print()\n",
    "    print(f\"Average loss on test set: {np.mean(losses):.3f}\")\n",
    "    print(f\"Accuracy on test set: {np.mean(acc):.3f}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}