04-predicting-academic-performance.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting Academic Performance\n",
    "## CSC2034\n",
    "### Cameron Trotter (c.trotter2@ncl.ac.uk)\n",
    "\n",
    "Now that we have played around with a few toy examples, let's take a look at tackling a real dataset. In this notebook we will work with the [Academic Performance dataset](https://www.kaggle.com/aljarah/xAPI-Edu-Data) from Kaggle. If after these sessions you want to expand your knowledge in your own time, Kaggle contains wide arrange of datasets and competitions for you to try. \n",
    "\n",
    "Unlike our synthetic dataset, the Academic Performance dataset contains more than two classes and a wide range of features. We will use this to help you understand the concept of overfitting, cross validation, and how to combat these through ensemble methods. \n",
    "\n",
    "### Google Colab Setup\n",
    "\n",
    "All of the notebooks you will be running in these lab sessions are designed to be ran using [Google Colab](https://colab.research.google.com/). For setup instructions, see this repo's README. \n",
    "\n",
    "In order to make things work on colab, we need to clone this repo and then (in another cell because colab dictates this...) move into the repo directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!git clone https://github.com/Trotts/csc2034-ds-demos.git"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.chdir('csc2034-ds-demos')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Importing Data\n",
    "\n",
    "First, let's import the Academic Performance dataset. For ease, I have downloaded the dataset and cleaned it up. In data science, cleaning data is an important but time consuming task, and as such I don't expect you to do it here; it would take far too long for the time we have. When working with textual datasets, it is highliy likely it will be provided to you as a CSV file. We can use [pandas](https://pandas.pydata.org/) to create a DataFrame object based on the CSV file we import. Most data science packages will be able to handle DataFrames as input. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "path = './data/academic_performance_clean.csv'\n",
    "\n",
    "dataset = pd.read_csv(path, header = 0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's examine the dataset to better understand the format and what it contains."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Number of examples: {dataset.shape[0]}\")\n",
    "print(f\"Number of features: {dataset.shape[1]}\")\n",
    "print(f\"List of features:\\n\\t{dataset.columns}\")\n",
    "\n",
    "print(f\"\\nExamining the first 5 entries in the dataset:\")\n",
    "display(dataset.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploratory Data Analysis (EDA)\n",
    "\n",
    "Before we can begin to train models we need to first perform some EDA. This allows us to better understand the data we have. As we can see fron examining the first few examples in the dataset, some features are numeric whilst some are categorical..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "categorical = ['Gender', 'Nationality', 'StageID',\n",
    "               'GradeID', 'SectionID', 'Topic',\n",
    "               'Semester', 'Relation', 'ParentAnsweringSurvey',\n",
    "               'ParentSchoolSatisfaction', 'StudentAbsenceDays']\n",
    "\n",
    "numerical = ['RaisedHands', 'VisitedResources', 'AnnouncementsView', 'Discussion']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot some of the categorical features using seaborn to understand the data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.figure(figsize = (10, 6))\n",
    "sns.countplot(x = \"Nationality\", data = dataset)\n",
    "plt.title(\"Count of different nationalities in the academic performance dataset\")\n",
    "plt.xlabel(\"Nationality\")\n",
    "plt.ylabel(\"Count\")\n",
    "plt.xticks(rotation=45) # Rotate the x labels so they don't overlap\n",
    "plt.show()\n",
    "plt.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this plot we can see our dataset is quite imbalanced. Kuwaiti and Jordanian nationals make up a significant proportion of the Nationality examples. As such, we now know we will have to handle this. \n",
    "\n",
    "Let's now take a look at numerical features. We can output summary statistics for them using `describe()`. The [docs](https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.describe.html) explain the output in detail should you need it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(dataset[numerical].describe())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Numerical values can also be plotted by examining their distribution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize = (30,6))\n",
    "sns.countplot(x = \"RaisedHands\", data = dataset)\n",
    "plt.title(\"Count of RaisedHands in the academic performance dataset\")\n",
    "plt.xlabel(\"Number of times student raised their hand\")\n",
    "plt.ylabel(\"Count\")\n",
    "plt.xticks(rotation=45) # Rotate the x labels so they don't overlap\n",
    "plt.show()\n",
    "plt.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using this dataset, we want to predict the `Class` of the student given the other features. In order to do this, we first need to know how many values `Class` can take, as well as its distribution. \n",
    "\n",
    "Task: Using seaborn, perform some EDA to understand the distribution and possible values `Class` can take. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the plot you have created, we can see there are three possible values `Class` can take; M (Medium), L (Low), or H (High). These tell us how good the student's overall academic performance was. \n",
    "\n",
    "Now we understand the categorical feature we are going to predict, let's plot a numerical feature, `RaisedHands`, aggregated by *another* categorical feature, `Gender`. This is a very powerful EDA tool to help us fully understand the data we have available. To do this, we create a `FacetGrid` based on `Class` and then fill this with barplots showing `RaisedHands` against `Gender`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grid = sns.FacetGrid(dataset, col=\"Class\")\n",
    "grid.map(sns.barplot, \"Gender\", \"RaisedHands\", order=[\"Male\", \"Female\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A pretty even `Gender` split for raising hands regardless of `Class`!\n",
    "\n",
    "Some of seaborn's plots take `kind` arguments which can change how the plotted data is displayed. For example `catplot` can take a wide variety of `kind` arguments, such as `box` or `violin`..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.figure(figsize = (10, 6))\n",
    "sns.catplot(x = \"Class\", y = \"RaisedHands\", data = dataset, kind = \"box\", hue = \"Gender\")\n",
    "plt.title(\"Boxplot of different Classes in the academic performance dataset per Gender\")\n",
    "plt.xlabel(\"Class\")\n",
    "plt.ylabel(\"Times hand raised\")\n",
    "plt.show()\n",
    "plt.close()\n",
    "\n",
    "plt.figure(figsize = (10, 6))\n",
    "sns.catplot(x = \"Class\", y = \"RaisedHands\", data = dataset, kind = \"violin\", hue = \"Gender\")\n",
    "plt.title(\"Violin plot of different Classes in the academic performance dataset per Gender\")\n",
    "plt.xlabel(\"Class\")\n",
    "plt.ylabel(\"Times hand raised\")\n",
    "plt.show()\n",
    "plt.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Task: Before continuing, use the examples above to build a wide variety of plots. Ensure each plot allows you to understand something different about your data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### One Hot Encoding\n",
    "\n",
    "When working with categorical data, we sometimes need to convert this to numeric to allow a classifier to train. Sometimes this isn't needed, but in most cases it will be. In reality this is usually a result of a constraint placed by the algorithm implementation rather than the algorithm theory. \n",
    "\n",
    "One of the easiest ways to convert our categorical features to numeric ones is through the use of One Hot Encoding. This creates *dummy features* for each categorical feature, with a new column created for each possible categorical value. The values in the new dummy feature are binary, denoting whether the categorical feature was of the value of the dummy.\n",
    "\n",
    "Let's take Gender as an example. This contains 2 possible values; Male or Female. Using One Hot Encoding we will create three new columns in our dataset with the following possible values:\n",
    "\n",
    "* Male: 0 or 1\n",
    "* Female: 0 or 1\n",
    "\n",
    "To add these columns to our dataset, we create the dummy vairables using `get_dummies`, concatinate the columns onto the dataset, and then drop the column we used to create the dummies.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dummies = pd.get_dummies(dataset[\"Gender\"])\n",
    "dataset_onehot = pd.concat((dataset, dummies), axis = 1) # axis = 1 == columns (0 == rows)\n",
    "dataset_onehot = dataset_onehot.drop(\"Gender\", axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's now examine the shape of our dataset like we did at the beginning. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Number of examples: {dataset_onehot.shape[0]}\")\n",
    "print(f\"Number of features: {dataset_onehot.shape[1]}\")\n",
    "print(f\"List of features:\\n\\t{dataset_onehot.columns}\")\n",
    "\n",
    "print(f\"\\nExamining the first 5 entries in the dataset:\")\n",
    "display(dataset_onehot.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have 17 columns instead of 16, as we have added 2 and removed 1. \n",
    "\n",
    "Task: Create dummy featues for all other categorical features in the dataset, except `Class`. You may find the use of an iterator useful here. Note that above, we created a new copy of the dataset called `dataset_onehot`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "remaining_categorical = ['Nationality', 'StageID',\n",
    "               'GradeID', 'SectionID', 'Topic',\n",
    "               'Semester', 'Relation', 'ParentAnsweringSurvey',\n",
    "               'ParentSchoolSatisfaction', 'StudentAbsenceDays']\n",
    "\n",
    "for feature in remaining_categorical:\n",
    "    dummies = pd.get_dummies(dataset_onehot[feature])\n",
    "    dataset_onehot = pd.concat((dataset_onehot, dummies), axis = 1)\n",
    "    dataset_onehot = dataset_onehot.drop(feature, axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the below checks. If any return False, take another look at the code you have written before continuing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Number of features check: {dataset_onehot.shape[1] == 59}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Handling our Predictive Feature\n",
    "\n",
    "As the feature we will be predicting, `Class`, is categorical we need to do something different when converting it to numeric. This is because our models will only predict one column as output. If we One Hot Encoded `Class` it would then be split over three columns, making it impossible to use as a label. \n",
    "\n",
    "To get around this we can set numeric lables for the different classes and iterate through the dataset converting as we go. Once converted, we drop `Class` from the One Hot Encoded dataset and replace it with the `Class` column we just created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class_numeric = {'L': 0, 'M': 1, 'H': 2}\n",
    "\n",
    "numeric_class_dataset = dataset.replace({'Class': class_numeric})\n",
    "\n",
    "dataset_onehot = dataset_onehot.drop('Class', axis = 1)\n",
    "dataset_onehot = pd.concat((dataset_onehot, numeric_class_dataset['Class']), axis = 1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the dataset we have now..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Number of examples: {dataset_onehot.shape[0]}\")\n",
    "print(f\"Number of features: {dataset_onehot.shape[1]}\")\n",
    "print(f\"List of features:\\n\\t{dataset_onehot.columns}\")\n",
    "\n",
    "print(f\"\\nExamining the first 5 entries in the dataset:\")\n",
    "display(dataset_onehot.head())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Splitting the dataset\n",
    "\n",
    "Before we go any further, let's ensure we have a train and test set. Before doing this, we need to drop `Class` from the dataset and assign it as our label class.\n",
    "\n",
    "Task: Split your dataset so that 33% of the data is in the test set. Ensure `random_state` is set to 10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "data = dataset_onehot.drop(['Class'], axis = 1).values\n",
    "labels = dataset['Class'].values\n",
    "\n",
    "data_train, data_test, labels_train, labels_test = train_test_split(data,\n",
    "                                                                    labels,\n",
    "                                                                    test_size = 0.33,\n",
    "                                                                    random_state = 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Oversampling the Dataset\n",
    "\n",
    "Now we have our dataset in a form useable to train a model on. Now, we need to make sure our dataset is balanced. Based on our EDA previously, we know that `Class` value Medium, now 1, is about twice as prevelant in the dataset as Low (0) or High (2). We can use SMOTE to oversample our train set to fix this. \n",
    "\n",
    "Task: Using what you saw in the previous notebook, use SMOTE to oversample our train set, balancing the number of classes. Set `random_state` equal to 10 to ensure reproducability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_train, labels_train = #..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the below checks. If any return False, take another look at the code you have written before continuing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "unique, counts = np.unique(labels_train, return_counts=True)\n",
    "\n",
    "print(f\"After SMOTE, number of class examples equal? {all(counts == 135)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Splitting and Scaling the Data\n",
    "\n",
    "Now that we have a balanced dataset, we need to split it into a training and test set and then scale both sets based on the train set's values.\n",
    "\n",
    "Task: Scale your dataset based on the training set. Again, ensure your `random_state` is set to 10."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = #..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the below checks. If any return False, take another look at the code you have written before continuing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Size of training set check: {data_train_scaled.shape == (405, 58)}\")\n",
    "print(f\"Size of test set check: {data_test_scaled.shape == (159, 58)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training a Model\n",
    "\n",
    "Now we have a dataset ready to train a model, let's look at doing that! Let's start by looking at a non-linear SVM. \n",
    "\n",
    "Task: Using the previous notebooks to help you, create a non-linear SVM using the following hyperparameters. Train the model, and generate a test set accuracy score."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "C = 100\n",
    "kernel = 'poly'\n",
    "random_state = 10\n",
    "degree = 10\n",
    "coef0 = 9\n",
    "\n",
    "non_linear_svm = #..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the accuracy we achieve."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Test acc: {test_acc * 100}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the below checks. If any return False, take another look at the code you have written before continuing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Test set accuracy check: {test_acc * 100 == 75.47169811320755}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preventing Overfitting\n",
    "\n",
    "One big problem faced by data science models is that of overfitting. This occurs when the trained model has been unable to generalise well with the training data we have provided it. If this happens, our model will perform very well on the training set, but perform worse on the test set. You will go into this in more detail in your lectures.\n",
    "\n",
    "Let's check our SVM for overfitting. We can do this by predicting on the train set, generating an accuracy score, and comparing it to the test set accuracy."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "non_linear_svm_predictions = non_linear_svm.predict(data_train_scaled)\n",
    "train_acc = accuracy_score(labels_train, non_linear_svm_predictions)\n",
    "\n",
    "print(f\"Train acc: {train_acc * 100}%\")\n",
    "print(f\"Test acc: {test_acc * 100}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on these results, we are almost certainly overfitting. We can tell this because of the 100% training accuracy, but only 75% accuracy on the test set. This is likely our hyperparameters.\n",
    "\n",
    "#### Hyperparameter Tuning and Cross Validation\n",
    "\n",
    "In a previous notebook, I mentioned how hyparameters are values you need to set to allow your model to train. The values you set for these hyperparameters can greatly influence how well your model can generalise, preventing overfitting. But how do you know which hyperparameters to choose?\n",
    "\n",
    "To help us decide which hyperparameters to pass to our models, we can utilise a process known as hyperparameter tuning. The simplest form of this is known as a Grid Search. During a Grid Search, we provide a list of values to try out for each hyperparameter. In practise a subset of hyperparameters are used, especially for deep learning models - in my research a grid search on a deep learning model took just over 1 month to finish using 2 GPUs!\n",
    "\n",
    "When we perform hyperparameter tuning, we don't want to expose our model to the test set as this may impact generalisation and stops us being able to produce a final evaluation on unseen data. With large datasets, often the train set will be split further into an evaluation set to get around this. The train set will be used for training, hyperparameter searches will be evaluated against the evaluation set, and the test set will be reserved for final evaluation on unseen data. \n",
    "\n",
    "In our case however we have a small dataset. To allow us to still perform hyperparameter tuning we can utilise a concept known as Cross Validation. Again you will go into more detail in your lectures, but cross validation allows you to split your training set multiple times into N *folds*. Hyperparameter tuning is then performed on the combination of these folds. This gives us, in essence, both a training and validation set even though we have very little data to work with. \n",
    "\n",
    "Let's try some hyperparameter tuning using cross validation on our non-linear SVM and see if we can reduce our overfitting. Somewhat annoyingly, we now have a hyperparameter for our cross validation - the number of folds!\n",
    "\n",
    "Note this next code block might take a little while to run..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "number_of_folds = 5\n",
    "random_state = 10\n",
    "\n",
    "parameters_to_tune = [{'kernel': ['poly', 'rbf'],\n",
    "                      'C': [1, 10],\n",
    "                      'degree': [1, 40], \n",
    "                      'coef0': [1, 9]}]\n",
    "\n",
    "search = GridSearchCV(SVC(random_state = random_state), parameters_to_tune, cv = number_of_folds)\n",
    "search.fit(data_train_scaled, labels_train)\n",
    "\n",
    "print(f\"Best parameters set found: {search.best_params_}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have found a set of hyperparameters that the search believes will reduce overfitting and improve our test set accuracy the most! Let's train a model using the found parameters and check."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "non_linear_svm = SVC(kernel = search.best_params_['kernel'],\n",
    "                     degree = search.best_params_['degree'],\n",
    "                     C = search.best_params_['C'],\n",
    "                     coef0 = search.best_params_['coef0'], random_state = 10)\n",
    "non_linear_svm.fit(data_train_scaled, labels_train)\n",
    "\n",
    "non_linear_svm_predictions = non_linear_svm.predict(data_train_scaled)\n",
    "train_acc = accuracy_score(labels_train, non_linear_svm_predictions)\n",
    "\n",
    "non_linear_svm_predictions = non_linear_svm.predict(data_test_scaled)\n",
    "test_acc = accuracy_score(labels_test, non_linear_svm_predictions)\n",
    "\n",
    "print(f\"Train acc: {train_acc * 100}%\")\n",
    "print(f\"Test acc: {test_acc * 100}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now our training accuracy has dropped to 87% but our test accuracy has increased to nearl 79%. This indicates the hyperparameters found in the search have had the desired effect - our overfitting has been reduced! With a larger parameter search, we may be able to increase our test set acciracy even further. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Decision Trees and Random Forests\n",
    "\n",
    "Now let's take a look at training a decision tree on the data we have. \n",
    "\n",
    "Task: Using the transformed data, train a decision tree, and generate a train and test accuracy to allow for an overfitting check."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random_state = 10\n",
    "\n",
    "decision_tree = #...\n",
    "\n",
    "decision_tree_predictions = #...\n",
    "train_acc = #...\n",
    "\n",
    "decision_tree_predictions = #...\n",
    "test_acc = #...\n",
    "\n",
    "print(f\"Train acc: {train_acc * 100}%\")\n",
    "print(f\"Test acc: {test_acc * 100}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the below checks. If any return False, take another look at the code you have written before continuing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Train set accuracy check: {train_acc * 100 == 100.0}\")\n",
    "print(f\"Test set accuracy check: {test_acc * 100 == 71.0691823899371}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One again, we observe overfitting. In fact, decisions trees are so prone to overfitting it is rare to see them used on their own. One way to reduce their overfitting is to train multiple differnt decisions trees and then use these to vote on the final result. This is known as a random forest, as your model consists of an ensamble of randomly generated decision trees. Let's take a look at building one of these. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "random_forest = RandomForestClassifier(n_estimators=50, max_depth=50, random_state=10)\n",
    "random_forest.fit(data_train_scaled, labels_train)\n",
    "\n",
    "random_forest_predictions = random_forest.predict(data_train_scaled)\n",
    "train_acc = accuracy_score(labels_train, random_forest_predictions)\n",
    "\n",
    "random_forest_predictions = random_forest.predict(data_test_scaled)\n",
    "test_acc = accuracy_score(labels_test, random_forest_predictions)\n",
    "\n",
    "print(f\"Train acc: {train_acc * 100}%\")\n",
    "print(f\"Test acc: {test_acc * 100}%\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see better test accuracy here, but the model may still be overfitting. Let's look at performing a hyperparameter search and seeing if we can do better. \n",
    "\n",
    "Task: Using GridSearchCV, perform a 5-fold cross-validation on the Random Forest. Using the best parameters found, train another Random Forest and generate train and test set accuracies. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "random_state = 10\n",
    "number_of_folds = 5\n",
    "\n",
    "parameters_to_tune = [{'n_estimators': [10, 15],\n",
    "                      'max_depth': [5, 10]}]\n",
    "\n",
    "search = GridSearchCV(RandomForestClassifier(random_state = random_state), parameters_to_tune, cv = number_of_folds)\n",
    "search.fit(data_train_scaled, labels_train)\n",
    "\n",
    "print(f\"Best parameters set found: {search.best_params_}\")\n",
    "\n",
    "random_forest = RandomForestClassifier(n_estimators=search.best_params_['n_estimators'],\n",
    "                                       max_depth=search.best_params_['max_depth'],\n",
    "                                       random_state=10)\n",
    "random_forest.fit(data_train_scaled, labels_train)\n",
    "\n",
    "random_forest_predictions = random_forest.predict(data_train_scaled)\n",
    "train_acc = accuracy_score(labels_train, random_forest_predictions)\n",
    "\n",
    "random_forest_predictions = random_forest.predict(data_test_scaled)\n",
    "test_acc = accuracy_score(labels_test, random_forest_predictions)\n",
    "\n",
    "print(f\"Train acc: {train_acc * 100}%\")\n",
    "print(f\"Test acc: {test_acc * 100}%\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Run the below checks. If any return False, take another look at the code you have written before continuing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f\"Train set accuracy check: {train_acc * 100 == 89.38271604938272}\")\n",
    "print(f\"Test set accuracy check: {test_acc * 100 == 74.84276729559748}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After tuning, we can see our train and test set accuracies are now close together. Even though we observe a drop in training accuracy, our test accuracy has very slightly increased - suggesting we have prevented overfitting and produced a generalised model. Again, with a larger hyperparameter search we may be able to do better here. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}