Update neural_nets.ipynb (aimacode#617)

Author: antmarakis

neural_nets.ipynb

@@ -19,7 +19,9 @@
    },
    "outputs": [],
    "source": [
-    "from learning import *"
+    "from learning import *\n",
+    "\n",
+    "from notebook import psource, pseudocode"
    ]
   },
   {
@@ -56,18 +58,18 @@
     "\n",
     "After that we will create our neural network in the `network` function. This function will make the necessary connections between the input layer, hidden layer and output layer. With the network ready, we will use the `BackPropagationLearner` to train the weights of our network for the examples provided in the dataset.\n",
     "\n",
-    "The NeuralNetLearner returns the `predict` function, which can receive an example and feed-forward it into our network to generate a prediction."
+    "The NeuralNetLearner returns the `predict` function which, in short, can receive an example and feed-forward it into our network to generate a prediction.\n",
+    "\n",
+    "In more detail, the example values are first passed to the input layer and then they are passed through the rest of the layers. Each node calculates the dot product of its inputs and its weights, activates it and pushes it to the next layer. The final prediction is the node with the maximum value from the output layer."
    ]
   },
   {
    "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {
-    "collapsed": true
-   },
+   "execution_count": null,
+   "metadata": {},
    "outputs": [],
    "source": [
-    "%psource NeuralNetLearner"
+    "psource(NeuralNetLearner)"
    ]
   },
   {
@@ -101,6 +103,66 @@
     "We can use the same technique for the weights in the input layer as well. After we have the gradients for both weights, we use gradient descent to update the weights of the network."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Pseudocode"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/markdown": [
+       "### AIMA3e\n",
+       "__function__ BACK-PROP-LEARNING(_examples_, _network_) __returns__ a neural network  \n",
+       " __inputs__ _examples_, a set of examples, each with input vector __x__ and output vector __y__  \n",
+       "&emsp;&emsp;&emsp;&emsp;_network_, a multilayer network with _L_ layers, weights _w<sub>i,j</sub>_, activation function _g_  \n",
+       "&emsp;__local variables__: &Delta;, a vector of errors, indexed by network node  \n",
+       "\n",
+       "&emsp;__repeat__  \n",
+       "&emsp;&emsp;&emsp;__for each__ weight _w<sub>i,j</sub>_ in _network_ __do__  \n",
+       "&emsp;&emsp;&emsp;&emsp;&emsp;_w<sub>i,j</sub>_ &larr; a small random number  \n",
+       "&emsp;&emsp;&emsp;__for each__ example (__x__, __y__) __in__ _examples_ __do__  \n",
+       "&emsp;&emsp;&emsp;&emsp;&emsp;/\\* _Propagate the inputs forward to compute the outputs_ \\*/  \n",
+       "&emsp;&emsp;&emsp;&emsp;&emsp;__for each__ node _i_ in the input layer __do__  \n",
+       "&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;_a<sub>i</sub>_ &larr; _x<sub>i</sub>_  \n",
+       "&emsp;&emsp;&emsp;&emsp;&emsp;__for__ _l_ = 2 __to__ _L_ __do__  \n",
+       "&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;__for each__ node _j_ in layer _l_ __do__  \n",
+       "&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;_in<sub>j</sub>_ &larr; &Sigma;<sub>_i_</sub> _w<sub>i,j</sub>_ _a<sub>i</sub>_  \n",
+       "&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;_a<sub>j</sub>_ &larr; _g_(_in<sub>j</sub>_)  \n",
+       "&emsp;&emsp;&emsp;&emsp;&emsp;/\\* _Propagate deltas backward from output layer to input layer_ \\*/  \n",
+       "&emsp;&emsp;&emsp;&emsp;&emsp;__for each__ node _j_ in the output layer __do__  \n",
+       "&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&Delta;\\[_j_\\] &larr; _g_&prime;(_in<sub>j</sub>_) &times; (_y<sub>i</sub>_ &minus; _a<sub>j</sub>_)  \n",
+       "&emsp;&emsp;&emsp;&emsp;&emsp;__for__ _l_ = _L_ &minus; 1 __to__ 1 __do__  \n",
+       "&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;__for each__ node _i_ in layer _l_ __do__  \n",
+       "&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&Delta;\\[_i_\\] &larr; _g_&prime;(_in<sub>i</sub>_) &Sigma;<sub>_j_</sub> _w<sub>i,j</sub>_ &Delta;\\[_j_\\]  \n",
+       "&emsp;&emsp;&emsp;&emsp;&emsp;/\\* _Update every weight in network using deltas_ \\*/  \n",
+       "&emsp;&emsp;&emsp;&emsp;&emsp;__for each__ weight _w<sub>i,j</sub>_ in _network_ __do__  \n",
+       "&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;&emsp;_w<sub>i,j</sub>_ &larr; _w<sub>i,j</sub>_ &plus; _&alpha;_ &times; _a<sub>i</sub>_ &times; &Delta;\\[_j_\\]  \n",
+       " &emsp;__until__ some stopping criterion is satisfied  \n",
+       " &emsp;__return__ _network_  \n",
+       "\n",
+       "---\n",
+       "__Figure ??__ The back\\-propagation algorithm for learning in multilayer networks."
+      ],
+      "text/plain": [
+       "<IPython.core.display.Markdown object>"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "pseudocode('Back-Prop-Learning')"
+   ]
+  },
   {
    "cell_type": "markdown",
    "metadata": {},
@@ -112,11 +174,11 @@
   },
   {
    "cell_type": "code",
-   "execution_count": 3,
+   "execution_count": null,
    "metadata": {},
    "outputs": [],
    "source": [
-    "%psource BackPropagationLearner"
+    "psource(BackPropagationLearner)"
    ]
   },
   {