lesson-06-part-02.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Connect Intensive - Machine Learning Nanodegree\n",
    "# Lesson 06: Clustering and Dimensionality Reduction\n",
    "# Part 02: PCA Mini Project\n",
    "\n",
    "## Objectives\n",
    "  - Perform Principal Component Analysis (PCA) on a large set of features to explain as much of the variance as possible in the data using a smaller set of features (reducing the dimensionality).\n",
    "  - Recognize differences between `train_test_split()` and `StratifiedShuffleSplit()` when creating training and testing sets.\n",
    "  - Introduce the `class_weight` parameter for `SVC()`, to see how correctly predicting targets from a smaller class size can be weighted more heavily.\n",
    "  - Visualize the eigenfaces (orthonormal basis of components) that result from PCA.\n",
    "  \n",
    "## Prerequisites\n",
    "  - You should have the following python packages installed:\n",
    "    - [matplotlib](http://matplotlib.org/index.html)\n",
    "    - [numpy](http://www.scipy.org/scipylib/download.html)\n",
    "    - [sklearn](http://scikit-learn.org/stable/install.html)\n",
    "  - For context, you should also have watched Sebastian & Katie's Udacity video lectures on PCA prior to working through this notebook.\n",
    "    \n",
    "## Acknowledgements\n",
    "  - The content of this notebook builds on the code from [pca/eigenfaces.py](https://github.com/udacity/ud120-projects/blob/master/pca/eigenfaces.py) from the [**ud120-projects**](https://github.com/udacity/ud120-projects) repo, and derives from [an eigenfaces/SVM example in the sklearn documentation](http://scikit-learn.org/stable/auto_examples/applications/face_recognition.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## PCA Mini-Project\n",
    "\n",
    "This Jupyter Notebook is intended to provide a friendly guide through the \"PCA Mini-Project\" lesson... but if you're feeling pretty confident about your Python skills, consider going off-script! Try to work through the lesson on your own -- you may encounter some snags, and you can always refer back to this Notebook if you need a little push forward.\n",
    "\n",
    "Hopefully you have watched Sebastian and Katie's video lectures on Principal Component Analysis -- if you haven't, I recommend catching up on those video lectures before tackling this project. That way, you will get the context of how and why we are using PCA to reduce the dimensionality of our dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Labeled Faces in the Wild\n",
    "\n",
    "The dataset for this mini-project comes from [\"Labeled Faces in the Wild\"](http://vis-www.cs.umass.edu/lfw/) or LFW, a database of more than 13,000 face photographs designed for studying the problem of unconstrained face recognition. We are going to use the **funneled** dataset, which comes from the following paper:\n",
    "\n",
    "> Gary B. Huang, Vidit Jain, and [Erik Learned-Miller](http://people.cs.umass.edu/~elm/).  \n",
    "[**Unsupervised Joint Alignment of Complex Images**](http://vis-www.cs.umass.edu/papers/iccv07alignment.pdf)  \n",
    "*International Conference on Computer Vision (ICCV), 2007.*\n",
    "\n",
    "If you're curious, section 2 of the paper (Background) describes the preprocessing steps of [congealing](http://vis-www.cs.umass.edu/congeal.html) and funneling to align images in the dataset.\n",
    "\n",
    "The original source of this PCA demo comes from the [scikit-learn documentation](http://scikit-learn.org/stable/auto_examples/applications/face_recognition.html).\n",
    "\n",
    "**Run** the cell below to import useful libraries and modules for this Jupyter notebook. We will use quite a few different functions from `sklearn` -- some we've seen before and others that are new."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "from time import time\n",
    "import logging\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.model_selection import StratifiedShuffleSplit\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.datasets import fetch_lfw_people\n",
    "from sklearn.metrics import classification_report\n",
    "from sklearn.metrics import confusion_matrix\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "# Display progress logs on stdout\n",
    "# (useful to monitor progress while downloading LFW data)\n",
    "logging.basicConfig(level=logging.INFO, format='%(asctime)s %(message)s')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Run** the cell below to download the data using [the `sklearn` function `fetch_lfw_people()`](http://scikit-learn.org/stable/modules/generated/sklearn.datasets.fetch_lfw_people.html). The LFW data is ~200MB, so the download may take a while. The data will be loaded into the data dictionary `lfw_people`. The dictionary contains several items:\n",
    "  - `lfw_people.DESCR`: A string -- a short description of the data dictionary.\n",
    "  - `lfw_people.data`: A 2-dimensional numpy array (`dtype=float32`) with shape `(n_samples, n_features)`\n",
    "    - Each entry in the array is a value ranging from 0.0 to 255.0, denoting the 8-bit [grayscale](https://en.wikipedia.org/wiki/Grayscale) value of each pixel.\n",
    "    - `n_samples` is the total number of images, the instances/inputs in the dataset.\n",
    "    - `n_features` is the number of pixels per image, the features of the dataset.\n",
    "  - `lfw_people.images`: A 3-dimensional numpy array (`dtype=float32`) with shape `(n_samples, height, width)`.\n",
    "    - `images` is just `data` reshaped, so that `height` $\\times$ `width` = `n_features`\n",
    "  - `lfw_people.target` : A 1-dimensional numpy array (`dtype=int64`) of length `n_samples`, the labels of each image\n",
    "  - `lfw_people.target_names`: A 1-dimensional numpy array of length `n_classes`\n",
    "  \n",
    "**Notes:** Sometimes [the pillow module](https://python-pillow.org/) (which is being used in this example) can cause trouble with this notebook. If you get an error related to the `fetch_lfw_people()` command, try the following:\n",
    "\n",
    "``pip install --upgrade PILLOW``\n",
    "\n",
    "If you run into a different error, note that there are changes in one of the parameters for the SVC function called on line 94 of pca/eigenfaces.py. For the `class_weight` parameter, the argument string \"auto\" is a valid value for sklearn version 0.16 and prior, but will be depreciated by 0.19. If you are running sklearn version 0.17 or later, the expected argument string should be \"balanced\". If you get an error or warning while running the [**ud120-projects**](https://github.com/udacity/ud120-projects) code in [pca/eigenfaces.py](https://github.com/udacity/ud120-projects/blob/master/pca/eigenfaces.py), make sure that you have the correct argument on line 98 of that file, that matches your installed version of sklearn."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Download the data, if not already on disk, and load it as numpy arrays\n",
    "lfw_people = fetch_lfw_people(min_faces_per_person=70, resize=0.4)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Above, in the call to `fetch_lfw_people()`, we set the parameter `min_faces_per_person` to 70. This parameter ensures that the extracted dataset will only retain pictures of people that have at least 70 different pictures. The `resize` parameter of 0.4 scales each image to 40% of its original size, so that the dataset takes up less working memory.\n",
    "\n",
    "**Run** the cell below to see the names of the people in the `lfw_people` dataset, with their corresponding target IDs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Print the indices and names of the people in the lfw_people dataset\n",
    "for idx, name in enumerate(lfw_people.target_names):\n",
    "    print(\"{}: {}\".format(idx,name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's continue exploring the `lfw_people` data dictionary! **Run** the cell below to extract the shape information (number of samples, number of features, image height and width) from `lfw_people.images` and `lfw_people.data`. You will see a summary of the dataset: the numbers of samples (images), features (pixels), and classes (targets). You will also see the number of images per target in the dataset -- note that we have many more images of certain targets!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# introspect the images arrays to find the shapes (for plotting)\n",
    "n_samples, h, w = lfw_people.images.shape\n",
    "\n",
    "# for machine learning we use the 2-dimensional data directly\n",
    "# (relative pixel positions info will be ignored by this model)\n",
    "X = lfw_people.data\n",
    "n_features = X.shape[1]\n",
    "\n",
    "# the label to predict is the target ID of the person\n",
    "y = lfw_people.target\n",
    "target_names = lfw_people.target_names\n",
    "n_classes = target_names.shape[0]\n",
    "\n",
    "# print a summary of the total dataset size\n",
    "print(\"Total dataset size:\")\n",
    "print(\"  n_samples  : {:>4}\".format(n_samples))\n",
    "print(\"  n_features : {:>4}\".format(n_features))\n",
    "print(\"  n_classes  : {:>4}\\n\".format(n_classes))\n",
    "\n",
    "# print the number of images for each target in the dataset\n",
    "print(\"Images per Target:\")\n",
    "for idx, name in enumerate(lfw_people.target_names):\n",
    "    print(\"  {:<18}: {:>3} images\".format(name,np.sum(lfw_people.target == idx)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Run** the cell below to split the data into training and testing (validation) sets. Note that we choose to use the function [`train_test_split()`](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) instead of [`StratifiedShuffleSplit()`](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.StratifiedShuffleSplit.html), so there's no guarantee that our training and testing sets will have the same proportion of each target or class. This cell will check to see how evenly `train_test_split()` partitions the data between training and testing set for each target. As a comparison, we also split the data using `StratifiedShuffleSplit()` and check the percentages of each target in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Split into a training and testing set using train_test_split\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)\n",
    "\n",
    "# Summarize the proportions of each label in the training and testing sets\n",
    "print(\"train_test_split() results:\")\n",
    "print(\"{:>28}{:>8}\".format(\"train\",\"test\"))\n",
    "for idx, name in enumerate(lfw_people.target_names):\n",
    "    print(\"  {:<18}: {:6.1f}% {:6.1f}%\"\\\n",
    "          .format(name,\\\n",
    "                  np.sum(y_train == idx) * 100.0 / len(y_train),\\\n",
    "                  np.sum(y_test == idx) * 100.0 / len(y_test)))\n",
    "    \n",
    "# Compare to training and testing set using StratifiedShuffleSplit()\n",
    "sss = StratifiedShuffleSplit(n_splits=1, test_size=0.25, random_state=42)\n",
    "train_index_sss, test_index_sss = sss.split(X, y).next()\n",
    "X_train_sss, X_test_sss = X[train_index_sss], X[test_index_sss]\n",
    "y_train_sss, y_test_sss = y[train_index_sss], y[test_index_sss]\n",
    "\n",
    "# Summarize the proportions of each label in the training and testing sets\n",
    "print(\"\\nStratifiedShuffleSplit() comparison:\")\n",
    "print(\"{:>28}{:>8}\".format(\"train\",\"test\"))\n",
    "for idx, name in enumerate(lfw_people.target_names):\n",
    "    print(\"  {:<18}: {:6.1f}% {:6.1f}%\"\\\n",
    "          .format(name,\\\n",
    "                  np.sum(y_train_sss == idx) * 100.0 / len(y_train_sss),\\\n",
    "                  np.sum(y_test_sss == idx) * 100.0 / len(y_test_sss)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the results of `train_test_split()`, we see that sometimes a label is slightly over-represented in the training set (*e.g.* for Ariel Sharon or Donald Rumsfeld). In contrast, the `StratifiedShuffleSplit()` method aims to match the representation of each label in the training and testing set as closely as possible. However, for the most part, there is not a significant difference in each label's representation when using `train_test_split()` -- so we'll proceed with the `train_test_split()` splits."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Principal Component Analysis\n",
    "\n",
    "The goal of [Principal Component Analysis (PCA)](https://en.wikipedia.org/wiki/Principal_component_analysis) is to describe the maximal amount of variance in a dataset by the fewest number of composite features. PCA as a concept is a lot easier to understand when seen, so for cool interactive visualizations on Principal Component Analysis, [check this out](http://setosa.io/ev/principal-component-analysis/) -- you can drag points in a dataset around in a plane and see the resulting axes for PCA on that dataset!\n",
    "\n",
    "**Run** the cell below to perform [Principal Component Analysis with `sklearn`](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html) on the training set of images to find the *eigenfaces*, or the orthonormal basis resulting from PCA.\n",
    "\n",
    "**Note:** There's a little background on [Singular Value Decomposition (SVD)](https://en.wikipedia.org/wiki/Singular_value_decomposition) and [Randomized SVD](https://research.facebook.com/blog/fast-randomized-svd/) that may be useful or interesting to read here... but it's not necessary to fully understand SVD to appreciate the results of PCA."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Compute a PCA (eigenfaces) on the face dataset\n",
    "# (here, we treat the images as an unlabeled dataset):\n",
    "# unsupervised feature extraction / dimensionality reduction\n",
    "\n",
    "# How many components from PCA to keep?\n",
    "n_components = 150\n",
    "\n",
    "# Perform PCA on the training data set, keep track of elapsed time.\n",
    "print(\"Extracting the top {} eigenfaces from {} faces\".format(n_components, X_train.shape[0]))\n",
    "t0 = time()\n",
    "\n",
    "# NOTE: in sklearn v0.17 and earlier, there was a separate method,\n",
    "#   sklearn.decomposition.RandomizedPCA()\n",
    "# that performed randomized Singular Value Decomposition (SVD)\n",
    "#\n",
    "# For sklearn v0.18 onward, the parameter svd_solver=\"randomized\"\n",
    "# in PCA() will do this -- no need for a separate method.\n",
    "pca = PCA(n_components=n_components, svd_solver='randomized',\\\n",
    "          whiten=True).fit(X_train)\n",
    "print(\"done in {:0.3f}s\".format(time() - t0))\n",
    "\n",
    "# Reshape the PCA components based on the image dimensions.\n",
    "eigenfaces = pca.components_.reshape((n_components, h, w))\n",
    "\n",
    "\n",
    "print(\"Projecting the input data on the eigenfaces orthonormal basis\")\n",
    "t0 = time()\n",
    "X_train_pca = pca.transform(X_train)\n",
    "X_test_pca = pca.transform(X_test)\n",
    "print(\"done in {:0.3f}s\".format(time() - t0))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Explained Variance Ratio\n",
    "\n",
    "Now that we've conducted PCA on the training dataset, we can see how much of the variance in the dataset is explained by each eigenface (or each component of the orthonormal basis). The components are ordered by ratio of explained variance in descending order, and after conducting PCA this information is stored in the attribute `pca.explained_variance_ratio_`. **Run** the cell below to print the explained variance percentage from the first five eigenfaces, and the total explained variance percentage of all 150 eigenfaces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "for idx, var in enumerate(pca.explained_variance_ratio_[:5]):\n",
    "    print(\"Eigenface {} explains {:5.2f}% of the variance.\".format(idx+1, var*100.0))\n",
    "    \n",
    "print(\"\\nIn total, the first {} eigenfaces explain {:5.2f}% of the variance.\"\\\n",
    "      .format(len(pca.explained_variance_ratio_),\\\n",
    "              100.0*np.sum(pca.explained_variance_ratio_)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Refining the accuracy score:  `class_weight`\n",
    "\n",
    "We saw earlier in this notebook that the different targets or classes have different representation, so [the accuracy score](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.accuracy_score.html) might not be the best measure of how good a predictive model is. For example, a predictive model where the prediction is always \"Ariel Sharon\" would have an accuracy score of about 6% on our dataset, while a predictive model where the prediction is always \"George W Bush\" would have an accuracy score of about 41% on our dataset -- but the second model isn't necessarily any better! To deal with this problem, we can adjust the `class_weight` parameter in `SVC()`, so that the model score improves by different amounts depending on the class of the that targets are accurately classified. For a more detailed explanation of the `class_weight` parameter, you could [read the `SVC()` documentation](http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html), or check out [this stackoverflow post](http://stackoverflow.com/questions/30972029/how-does-the-class-weight-parameter-in-scikit-learn-work) -- I found the response by Andreas Mueller helpful.\n",
    "\n",
    "The cell below has the formula for computing the weights when setting the `class_weight` parameter in `SVC()` to `\"balanced\"`. **Run** the cell below to display the weights for each class or target, along with the number of instances of each target. This cell shows that rarer classes or targets are weighted more heavily -- it's more important for the classifier to get those targets correct!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "weights = n_samples*1.0 / (n_classes * np.bincount(y_train))\n",
    "print(\"{:<23}{:<8}{:<6}\".format(\"Target Name\",\"Weight\",\"Count\"))\n",
    "for idx, name in enumerate(lfw_people.target_names):\n",
    "    print(\"  {:<18}: {:6.2f} {:>6}\".format(name, weights[idx], np.sum(y_train == idx)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we understand how the `class_weight` parameter adds weight to the rarer classes, we can use [the `GridSearchCV()` function](http://scikit-learn.org/stable/modules/generated/sklearn.model_selection.GridSearchCV.html) to find the optimal parameters for a Support Vector Machine (SVM) classifier. Here, we're using the radial basis function (RBF) kernel, and we're tuning the parameters `C` (hard-margin SVM when `C` is large vs. soft-margin SVM when `C` is small) and `gamma` (data points have a long-ranged influence when `gamma` is small, or a short-ranged influence when `gamma` is large). when For more details on SVM, you can check [lesson-05.ipynb](https://github.com/nickypie/ConnectIntensive/blob/master/lesson-05.ipynb) in the [ConnectIntensive](https://github.com/nickypie/ConnectIntensive) repo.\n",
    "\n",
    "**Run** the cell below to find the optimal values for the `C` and `gamma` parameters in the `SVC()` classifier,  using the eigenfaces (`X_train_pca`) as the features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Train a SVM classification model\n",
    "\n",
    "print(\"Fitting the classifier to the training set\\n\")\n",
    "t0 = time()\n",
    "param_grid = {'gamma' :[0.0001, 0.0005, 0.001, 0.005, 0.01, 0.1],\\\n",
    "              'C'     :[1e3, 5e3, 1e4, 5e4, 1e5]}\n",
    "\n",
    "# for sklearn version 0.16 or prior, the class_weight parameter value is 'auto'\n",
    "clf = GridSearchCV(SVC(kernel='rbf', class_weight='balanced'), param_grid)\n",
    "clf = clf.fit(X_train_pca, y_train)\n",
    "\n",
    "# Display the detailed results of the grid search + cross-validation:\n",
    "print (\"done in {:0.3f}s\\n\".format(time() - t0))\n",
    "print(\"Best parameters set found on development set:\\n\")\n",
    "print(clf.best_params_)\n",
    "print(\"\\nGrid scores on development set:\\n\")\n",
    "means = clf.cv_results_['mean_test_score']\n",
    "stds = clf.cv_results_['std_test_score']\n",
    "for mean, std, params in zip(means, stds, clf.cv_results_['params']):\n",
    "    print(\"%0.3f (+/-%0.03f) for %r\"\n",
    "          % (mean, std * 2, params))\n",
    "print(\" \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this optimal SVM classifier, let's actually make predictions on the test data! **Run** the cell below to make predictions on the test data. Be careful here! We trained the classifier using the *projections* of the training set features onto the *eigenfaces* (the orthonormal basis of transformed coordinates), so we must also make predictions using the projection of the test set features onto eigenfaces -- the classifier won't work on the original (pre-transformed) features.\n",
    "\n",
    "We will also see a **classification report** and a **confusion matrix**. The classification report summarizes the following metrics for each class or target:\n",
    "  - [**precision**](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html): Precision `p` is the ratio `tp / (tp + fp)` where `tp` is the number of **true positives** and `fp` the number of **false positives**. For example, if I'm trying to identify George W Bush, a true positive would be when the classifier correctly identifies an image of George W Bush, and a false positive would be when the classifier identifies an image of someone else as George W Bush.\n",
    "  - [**recall**](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.recall_score.html): Recall `r` is the ratio `tp / (tp + fn)` where `tp` is the number of **true positives** and `fn` the number of **false negatives**. For example, if I'm trying to identify George W Bush, a true positive would be when the classifier correctly identifies an image of George W Bush, and a false negative would be when the classifier identifies an image of George W Bush as someone else.\n",
    "  - [**f1-score**](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html): The f1-score `f1` is derived from the precision `p` and recall `r`: `f1 = 2*p*r / (p+r)`\n",
    "  - [**support**](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_recall_fscore_support.html): The support is the number of *true occurrences* of each class.\n",
    "  \n",
    "The [**confusion matrix**](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html) is a 2-d array that summarizes the predictions versus the true values.\n",
    "  - *Rows* of the confusion matrix correspond to **true values**.\n",
    "  - *Columns* of the confusion matrix correspond to **predictions**.\n",
    "  - *Correct* predictions lie along the **main diagonal**, while *incorrect* predictions are the **off-diagonal** entries.\n",
    "  - The sum of each row gives the **support** value for that class or label.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(\"Predicting the people names on the testing set\")\n",
    "t0 = time()\n",
    "y_pred = clf.predict(X_test_pca)\n",
    "print (\"done in {:0.3f}s\\n\".format(time() - t0))\n",
    "\n",
    "print(classification_report(y_test, y_pred, target_names=target_names))\n",
    "\n",
    "print(\"Confusion Matrix:\")\n",
    "print(confusion_matrix(y_test, y_pred, labels=range(n_classes)))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What does an eigenface look like?\n",
    "\n",
    "Now we're going to define a couple helper functions, `plot_gallery()` and `title()`, to help us see some sample faces from the test set, and visualize the first few eigenfaces to see what features explain the greatest amount of variance in the dataset.\n",
    "\n",
    "**Run** the cell below to define the two helper functions `plot_gallery()` and `title()` -- do you follow what each line of code does in the two helper functions?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def plot_gallery(images, titles, h, w, n_row=3, n_col=5):\n",
    "    \"\"\"Helper function to plot a gallery of portraits\"\"\"\n",
    "    plt.figure(figsize=(1.8 * n_col, 2.4 * n_row))\n",
    "    plt.subplots_adjust(bottom=0, left=.01, right=.99, top=.90, hspace=.35)\n",
    "    for i in range(n_row * n_col):\n",
    "        plt.subplot(n_row, n_col, i + 1)\n",
    "        plt.imshow(images[i].reshape((h, w)), cmap=plt.cm.gray)\n",
    "        plt.title(titles[i], size=12)\n",
    "        plt.xticks(())\n",
    "        plt.yticks(())\n",
    "\n",
    "\n",
    "def title(y_pred, y_test, target_names, i):\n",
    "    pred_name = target_names[y_pred[i]].rsplit(' ', 1)[-1]\n",
    "    true_name = target_names[y_test[i]].rsplit(' ', 1)[-1]\n",
    "    return 'predicted: {}\\ntrue:      {}'.format(pred_name, true_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's put the helper functions to work! **Run** the cell below to visualize the first few faces from the test set, labeled with the predicted and actual classes. How did our optimal classifier do on these faces?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# plot the result of the prediction on a portion of the test set\n",
    "\n",
    "prediction_titles = [title(y_pred, y_test, target_names, i)\n",
    "                         for i in range(y_pred.shape[0])]\n",
    "\n",
    "plot_gallery(X_test, prediction_titles, h, w)\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now this part is a little spooky (maybe we should have covered this lesson during Halloween). **Run** the cell below to visualize the first few eigenfaces (components of the orthonormal basis that maximally explains the variance in the dataset). You should see some ghostly apparitions that resemble faces, all arising from PCA!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# plot the gallery of the most significative eigenfaces\n",
    "\n",
    "eigenface_titles = [\"eigenface {}\".format(i) for i in range(eigenfaces.shape[0])]\n",
    "plot_gallery(eigenfaces, eigenface_titles, h, w)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Time to tackle the quizzes!\n",
    "\n",
    "Now that you've worked through this notebook, you should have answers to the some of the quizzes in the PCA mini-project. However, you may need to write additional code for some of the questions we've yet to answer. If you get stuck, a good place to start is the `sklearn` documentation. Good luck!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}