lime.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## License \n",
    "\n",
    "Copyright 2017 - 2020 Patrick Hall and the H2O.ai team\n",
    "\n",
    "Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "you may not use this file except in compliance with the License.\n",
    "You may obtain a copy of the License at\n",
    "\n",
    "    http://www.apache.org/licenses/LICENSE-2.0\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software\n",
    "distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "See the License for the specific language governing permissions and\n",
    "limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**DISCLAIMER:** This notebook is not legal compliance advice."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Explain Your Predictive Models to Business Stakeholders using LIME with Python and H2O\n",
    "#### Describing complex models and generating reason codes with Local Interpretable Model-agnostic Explanations (LIME) and LIME-variants\n",
    "\n",
    "Local Interpretable Model-agnostic Explanations (LIME) shed light on how almost any machine learning model makes decisions for specific rows of data. LIME builds local linear surrogate models around observations of interest and leverages the highly interpretable properties of linear models to increase transparency and accountability for the corresponding model predictions. In this notebook, an h2o GBM is trained on the UCI credit card default data and then predictions for a highly risky customer are explained using linear model coefficients and LIME-derived reason codes. The notebook concludes by introducing a variant of LIME that is easier to execute on new data and that can be analyzed alongside observed (i.e., not simulated) data.\n",
    "\n",
    "**Note**: As of the h2o 3.24 \"Yates\" release, Shapley values are supported in h2o. Shapley values can be used in place of or along with LIME. To see Shapley values for an h2o GBM in action please see: https://github.com/jphall663/interpretable_machine_learning_with_python/blob/master/dia.ipynb."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Python imports\n",
    "In general, NumPy and Pandas will be used for data manipulation purposes and h2o will be used for modeling tasks. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# h2o Python API with specific classes\n",
    "import h2o \n",
    "from h2o.estimators.glm import H2OGeneralizedLinearEstimator # for LIME\n",
    "from h2o.grid.grid_search import H2OGridSearch               # for LIME\n",
    "from h2o.estimators.gbm import H2OGradientBoostingEstimator  # for GBM\n",
    "\n",
    "\n",
    "import operator # for sorting dictionaries\n",
    "\n",
    "import numpy as np   # array, vector, matrix calculations\n",
    "import pandas as pd  # DataFrame handling\n",
    "\n",
    "# display plots in notebook\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Start h2o\n",
    "H2o is both a library and a server. The machine learning algorithms in the library take advantage of the multithreaded and distributed architecture provided by the server to train machine learning algorithms extremely efficiently. The API for the library was imported above in cell 1, but the server still needs to be started."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Checking whether there is an H2O instance running at http://localhost:54321 ..... not found.\n",
      "Attempting to start a local H2O server...\n",
      "  Java Version: java version \"1.8.0_201\"; Java(TM) SE Runtime Environment (build 1.8.0_201-b09); Java HotSpot(TM) 64-Bit Server VM (build 25.201-b09, mixed mode)\n",
      "  Starting server from /home/patrickh/anaconda3/lib/python3.6/site-packages/h2o/backend/bin/h2o.jar\n",
      "  Ice root: /tmp/tmpq9cbjivf\n",
      "  JVM stdout: /tmp/tmpq9cbjivf/h2o_patrickh_started_from_python.out\n",
      "  JVM stderr: /tmp/tmpq9cbjivf/h2o_patrickh_started_from_python.err\n",
      "  Server is running at http://127.0.0.1:54321\n",
      "Connecting to H2O server at http://127.0.0.1:54321 ... successful.\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div style=\"overflow:auto\"><table style=\"width:50%\"><tr><td>H2O cluster uptime:</td>\n",
       "<td>01 secs</td></tr>\n",
       "<tr><td>H2O cluster timezone:</td>\n",
       "<td>America/New_York</td></tr>\n",
       "<tr><td>H2O data parsing timezone:</td>\n",
       "<td>UTC</td></tr>\n",
       "<tr><td>H2O cluster version:</td>\n",
       "<td>3.26.0.3</td></tr>\n",
       "<tr><td>H2O cluster version age:</td>\n",
       "<td>4 days </td></tr>\n",
       "<tr><td>H2O cluster name:</td>\n",
       "<td>H2O_from_python_patrickh_zub5an</td></tr>\n",
       "<tr><td>H2O cluster total nodes:</td>\n",
       "<td>1</td></tr>\n",
       "<tr><td>H2O cluster free memory:</td>\n",
       "<td>1.778 Gb</td></tr>\n",
       "<tr><td>H2O cluster total cores:</td>\n",
       "<td>8</td></tr>\n",
       "<tr><td>H2O cluster allowed cores:</td>\n",
       "<td>8</td></tr>\n",
       "<tr><td>H2O cluster status:</td>\n",
       "<td>accepting new members, healthy</td></tr>\n",
       "<tr><td>H2O connection url:</td>\n",
       "<td>http://127.0.0.1:54321</td></tr>\n",
       "<tr><td>H2O connection proxy:</td>\n",
       "<td>None</td></tr>\n",
       "<tr><td>H2O internal security:</td>\n",
       "<td>False</td></tr>\n",
       "<tr><td>H2O API Extensions:</td>\n",
       "<td>Amazon S3, XGBoost, Algos, AutoML, Core V3, Core V4</td></tr>\n",
       "<tr><td>Python version:</td>\n",
       "<td>3.6.4 final</td></tr></table></div>"
      ],
      "text/plain": [
       "--------------------------  ---------------------------------------------------\n",
       "H2O cluster uptime:         01 secs\n",
       "H2O cluster timezone:       America/New_York\n",
       "H2O data parsing timezone:  UTC\n",
       "H2O cluster version:        3.26.0.3\n",
       "H2O cluster version age:    4 days\n",
       "H2O cluster name:           H2O_from_python_patrickh_zub5an\n",
       "H2O cluster total nodes:    1\n",
       "H2O cluster free memory:    1.778 Gb\n",
       "H2O cluster total cores:    8\n",
       "H2O cluster allowed cores:  8\n",
       "H2O cluster status:         accepting new members, healthy\n",
       "H2O connection url:         http://127.0.0.1:54321\n",
       "H2O connection proxy:\n",
       "H2O internal security:      False\n",
       "H2O API Extensions:         Amazon S3, XGBoost, Algos, AutoML, Core V3, Core V4\n",
       "Python version:             3.6.4 final\n",
       "--------------------------  ---------------------------------------------------"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "h2o.init(max_mem_size='2G')       # start h2o\n",
    "h2o.remove_all()                  # remove any existing data structures from h2o memory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Download, explore, and prepare UCI credit card default data\n",
    "\n",
    "UCI credit card default data: https://archive.ics.uci.edu/ml/datasets/default+of+credit+card+clients\n",
    "\n",
    "The UCI credit card default data contains demographic and payment information about credit card customers in Taiwan in the year 2005. The data set contains 23 input variables: \n",
    "\n",
    "* **`LIMIT_BAL`**: Amount of given credit (NT dollar)\n",
    "* **`SEX`**: 1 = male; 2 = female\n",
    "* **`EDUCATION`**: 1 = graduate school; 2 = university; 3 = high school; 4 = others \n",
    "* **`MARRIAGE`**: 1 = married; 2 = single; 3 = others\n",
    "* **`AGE`**: Age in years \n",
    "* **`PAY_0`, `PAY_2` - `PAY_6`**: History of past payment; `PAY_0` = the repayment status in September, 2005; `PAY_2` = the repayment status in August, 2005; ...; `PAY_6` = the repayment status in April, 2005. The measurement scale for the repayment status is: -1 = pay duly; 1 = payment delay for one month; 2 = payment delay for two months; ...; 8 = payment delay for eight months; 9 = payment delay for nine months and above. \n",
    "* **`BILL_AMT1` - `BILL_AMT6`**: Amount of bill statement (NT dollar). `BILL_AMNT1` = amount of bill statement in September, 2005; `BILL_AMT2` = amount of bill statement in August, 2005; ...; `BILL_AMT6` = amount of bill statement in April, 2005. \n",
    "* **`PAY_AMT1` - `PAY_AMT6`**: Amount of previous payment (NT dollar). `PAY_AMT1` = amount paid in September, 2005; `PAY_AMT2` = amount paid in August, 2005; ...; `PAY_AMT6` = amount paid in April, 2005. \n",
    "\n",
    "These 23 input variables are used to predict the target variable, whether or not a customer defaulted on their credit card bill in late 2005.\n",
    "\n",
    "Because h2o accepts both numeric and character inputs, some variables will be recoded into more transparent character values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Import data and clean\n",
    "The credit card default data is available as an `.xls` file. Pandas reads `.xls` files automatically, so it's used to load the credit card default data and give the prediction target a shorter name: `DEFAULT_NEXT_MONTH`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import XLS file\n",
    "path = 'default_of_credit_card_clients.xls'\n",
    "data = pd.read_excel(path,\n",
    "                     skiprows=1)\n",
    "\n",
    "# remove spaces from target column name \n",
    "data = data.rename(columns={'default payment next month': 'DEFAULT_NEXT_MONTH'}) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Assign modeling roles\n",
    "The shorthand name `y` is assigned to the prediction target. `X` is assigned to all other input variables in the credit card default data except the row indentifier, `ID`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y = DEFAULT_NEXT_MONTH\n",
      "X = ['LIMIT_BAL', 'SEX', 'EDUCATION', 'MARRIAGE', 'AGE', 'PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6', 'BILL_AMT1', 'BILL_AMT2', 'BILL_AMT3', 'BILL_AMT4', 'BILL_AMT5', 'BILL_AMT6', 'PAY_AMT1', 'PAY_AMT2', 'PAY_AMT3', 'PAY_AMT4', 'PAY_AMT5', 'PAY_AMT6']\n"
     ]
    }
   ],
   "source": [
    "# assign target and inputs for GBM\n",
    "y = 'DEFAULT_NEXT_MONTH'\n",
    "X = [name for name in data.columns if name not in [y, 'ID']]\n",
    "print('y =', y)\n",
    "print('X =', X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Helper function for recoding values in the UCI credict card default data\n",
    "This simple function maps longer, more understandable character string values from the UCI credit card default data dictionary to the original integer values of the input variables found in the dataset. These character values can be used directly in h2o decision tree models, and the function returns the original Pandas DataFrame as an h2o object, an H2OFrame. H2o models cannot run on Pandas DataFrames. They require H2OFrames."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n"
     ]
    }
   ],
   "source": [
    "def recode_cc_data(frame):\n",
    "    \n",
    "    \"\"\" Recodes numeric categorical variables into categorical character variables\n",
    "    with more transparent values. \n",
    "    \n",
    "    Args:\n",
    "        frame: Pandas DataFrame version of UCI credit card default data.\n",
    "        \n",
    "    Returns: \n",
    "        H2OFrame with recoded values.\n",
    "        \n",
    "    \"\"\"\n",
    "    \n",
    "    # define recoded values\n",
    "    sex_dict = {1:'male', 2:'female'}\n",
    "    education_dict = {0:'other', 1:'graduate school', 2:'university', 3:'high school', \n",
    "                      4:'other', 5:'other', 6:'other'}\n",
    "    marriage_dict = {0:'other', 1:'married', 2:'single', 3:'divorced'}\n",
    "    pay_dict = {-2:'no consumption', -1:'pay duly', 0:'use of revolving credit', 1:'1 month delay', \n",
    "                2:'2 month delay', 3:'3 month delay', 4:'4 month delay', 5:'5 month delay', 6:'6 month delay', \n",
    "                7:'7 month delay', 8:'8 month delay', 9:'9+ month delay'}\n",
    "    \n",
    "    # recode values using Pandas apply() and anonymous function\n",
    "    frame['SEX'] = frame['SEX'].apply(lambda i: sex_dict[i])\n",
    "    frame['EDUCATION'] = frame['EDUCATION'].apply(lambda i: education_dict[i])    \n",
    "    frame['MARRIAGE'] = frame['MARRIAGE'].apply(lambda i: marriage_dict[i]) \n",
    "    for name in frame.columns:\n",
    "        if name in ['PAY_0', 'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6']:\n",
    "            frame[name] = frame[name].apply(lambda i: pay_dict[i])            \n",
    "                \n",
    "    return h2o.H2OFrame(frame)\n",
    "\n",
    "data = recode_cc_data(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Ensure target is handled as a categorical variable\n",
    "In h2o, a numeric variable can be treated as numeric or categorical. The target variable `DEFAULT_NEXT_MONTH` takes on values of `0` or `1`. To ensure this numeric variable is treated as a categorical variable, the `asfactor()` function is used to explicitly declare that it is a categorical variable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "data[y] = data[y].asfactor() "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display descriptive statistics\n",
    "The h2o `describe()` function displays a brief description of the credit card default data. For the categorical input variables `LIMIT_BAL`, `SEX`, `EDUCATION`, `MARRIAGE`, and `PAY_0`-`PAY_6`, the new character values created above in cell 5 are visible. Basic descriptive statistics are displayed for numeric inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Rows:30000\n",
      "Cols:25\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th>       </th><th>ID               </th><th>LIMIT_BAL         </th><th>SEX   </th><th>EDUCATION      </th><th>MARRIAGE  </th><th>AGE              </th><th>PAY_0                  </th><th>PAY_2                  </th><th>PAY_3                  </th><th>PAY_4                  </th><th>PAY_5                  </th><th>PAY_6                  </th><th>BILL_AMT1        </th><th>BILL_AMT2        </th><th>BILL_AMT3        </th><th>BILL_AMT4        </th><th>BILL_AMT5        </th><th>BILL_AMT6        </th><th>PAY_AMT1          </th><th>PAY_AMT2          </th><th>PAY_AMT3          </th><th>PAY_AMT4          </th><th>PAY_AMT5          </th><th>PAY_AMT6          </th><th>DEFAULT_NEXT_MONTH  </th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td>type   </td><td>int              </td><td>int               </td><td>enum  </td><td>enum           </td><td>enum      </td><td>int              </td><td>enum                   </td><td>enum                   </td><td>enum                   </td><td>enum                   </td><td>enum                   </td><td>enum                   </td><td>int              </td><td>int              </td><td>int              </td><td>int              </td><td>int              </td><td>int              </td><td>int               </td><td>int               </td><td>int               </td><td>int               </td><td>int               </td><td>int               </td><td>enum                </td></tr>\n",
       "<tr><td>mins   </td><td>1.0              </td><td>10000.0           </td><td>      </td><td>               </td><td>          </td><td>21.0             </td><td>                       </td><td>                       </td><td>                       </td><td>                       </td><td>                       </td><td>                       </td><td>-165580.0        </td><td>-69777.0         </td><td>-157264.0        </td><td>-170000.0        </td><td>-81334.0         </td><td>-339603.0        </td><td>0.0               </td><td>0.0               </td><td>0.0               </td><td>0.0               </td><td>0.0               </td><td>0.0               </td><td>                    </td></tr>\n",
       "<tr><td>mean   </td><td>15000.5          </td><td>167484.32266666688</td><td>      </td><td>               </td><td>          </td><td>35.48549999999994</td><td>                       </td><td>                       </td><td>                       </td><td>                       </td><td>                       </td><td>                       </td><td>51223.33090000009</td><td>49179.07516666668</td><td>47013.15479999971</td><td>43262.9489666666 </td><td>40311.40096666653</td><td>38871.76039999991</td><td>5663.580500000014 </td><td>5921.16350000001  </td><td>5225.681500000005 </td><td>4826.076866666661 </td><td>4799.387633333302 </td><td>5215.502566666664 </td><td>                    </td></tr>\n",
       "<tr><td>maxs   </td><td>30000.0          </td><td>1000000.0         </td><td>      </td><td>               </td><td>          </td><td>79.0             </td><td>                       </td><td>                       </td><td>                       </td><td>                       </td><td>                       </td><td>                       </td><td>964511.0         </td><td>983931.0         </td><td>1664089.0        </td><td>891586.0         </td><td>927171.0         </td><td>961664.0         </td><td>873552.0          </td><td>1684259.0         </td><td>896040.0          </td><td>621000.0          </td><td>426529.0          </td><td>528666.0          </td><td>                    </td></tr>\n",
       "<tr><td>sigma  </td><td>8660.398374208891</td><td>129747.66156720225</td><td>      </td><td>               </td><td>          </td><td>9.21790406809016 </td><td>                       </td><td>                       </td><td>                       </td><td>                       </td><td>                       </td><td>                       </td><td>73635.86057552959</td><td>71173.76878252836</td><td>69349.38742703681</td><td>64332.85613391641</td><td>60797.1557702648 </td><td>59554.10753674574</td><td>16563.280354025763</td><td>23040.870402057226</td><td>17606.961469803115</td><td>15666.159744031993</td><td>15278.305679144793</td><td>17777.465775435332</td><td>                    </td></tr>\n",
       "<tr><td>zeros  </td><td>0                </td><td>0                 </td><td>      </td><td>               </td><td>          </td><td>0                </td><td>                       </td><td>                       </td><td>                       </td><td>                       </td><td>                       </td><td>                       </td><td>2008             </td><td>2506             </td><td>2870             </td><td>3195             </td><td>3506             </td><td>4020             </td><td>5249              </td><td>5396              </td><td>5968              </td><td>6408              </td><td>6703              </td><td>7173              </td><td>                    </td></tr>\n",
       "<tr><td>missing</td><td>0                </td><td>0                 </td><td>0     </td><td>0              </td><td>0         </td><td>0                </td><td>0                      </td><td>0                      </td><td>0                      </td><td>0                      </td><td>0                      </td><td>0                      </td><td>0                </td><td>0                </td><td>0                </td><td>0                </td><td>0                </td><td>0                </td><td>0                 </td><td>0                 </td><td>0                 </td><td>0                 </td><td>0                 </td><td>0                 </td><td>0                   </td></tr>\n",
       "<tr><td>0      </td><td>1.0              </td><td>20000.0           </td><td>female</td><td>university     </td><td>married   </td><td>24.0             </td><td>2 month delay          </td><td>2 month delay          </td><td>pay duly               </td><td>pay duly               </td><td>no consumption         </td><td>no consumption         </td><td>3913.0           </td><td>3102.0           </td><td>689.0            </td><td>0.0              </td><td>0.0              </td><td>0.0              </td><td>0.0               </td><td>689.0             </td><td>0.0               </td><td>0.0               </td><td>0.0               </td><td>0.0               </td><td>1                   </td></tr>\n",
       "<tr><td>1      </td><td>2.0              </td><td>120000.0          </td><td>female</td><td>university     </td><td>single    </td><td>26.0             </td><td>pay duly               </td><td>2 month delay          </td><td>use of revolving credit</td><td>use of revolving credit</td><td>use of revolving credit</td><td>2 month delay          </td><td>2682.0           </td><td>1725.0           </td><td>2682.0           </td><td>3272.0           </td><td>3455.0           </td><td>3261.0           </td><td>0.0               </td><td>1000.0            </td><td>1000.0            </td><td>1000.0            </td><td>0.0               </td><td>2000.0            </td><td>1                   </td></tr>\n",
       "<tr><td>2      </td><td>3.0              </td><td>90000.0           </td><td>female</td><td>university     </td><td>single    </td><td>34.0             </td><td>use of revolving credit</td><td>use of revolving credit</td><td>use of revolving credit</td><td>use of revolving credit</td><td>use of revolving credit</td><td>use of revolving credit</td><td>29239.0          </td><td>14027.0          </td><td>13559.0          </td><td>14331.0          </td><td>14948.0          </td><td>15549.0          </td><td>1518.0            </td><td>1500.0            </td><td>1000.0            </td><td>1000.0            </td><td>1000.0            </td><td>5000.0            </td><td>0                   </td></tr>\n",
       "<tr><td>3      </td><td>4.0              </td><td>50000.0           </td><td>female</td><td>university     </td><td>married   </td><td>37.0             </td><td>use of revolving credit</td><td>use of revolving credit</td><td>use of revolving credit</td><td>use of revolving credit</td><td>use of revolving credit</td><td>use of revolving credit</td><td>46990.0          </td><td>48233.0          </td><td>49291.0          </td><td>28314.0          </td><td>28959.0          </td><td>29547.0          </td><td>2000.0            </td><td>2019.0            </td><td>1200.0            </td><td>1100.0            </td><td>1069.0            </td><td>1000.0            </td><td>0                   </td></tr>\n",
       "<tr><td>4      </td><td>5.0              </td><td>50000.0           </td><td>male  </td><td>university     </td><td>married   </td><td>57.0             </td><td>pay duly               </td><td>use of revolving credit</td><td>pay duly               </td><td>use of revolving credit</td><td>use of revolving credit</td><td>use of revolving credit</td><td>8617.0           </td><td>5670.0           </td><td>35835.0          </td><td>20940.0          </td><td>19146.0          </td><td>19131.0          </td><td>2000.0            </td><td>36681.0           </td><td>10000.0           </td><td>9000.0            </td><td>689.0             </td><td>679.0             </td><td>0                   </td></tr>\n",
       "<tr><td>5      </td><td>6.0              </td><td>50000.0           </td><td>male  </td><td>graduate school</td><td>single    </td><td>37.0             </td><td>use of revolving credit</td><td>use of revolving credit</td><td>use of revolving credit</td><td>use of revolving credit</td><td>use of revolving credit</td><td>use of revolving credit</td><td>64400.0          </td><td>57069.0          </td><td>57608.0          </td><td>19394.0          </td><td>19619.0          </td><td>20024.0          </td><td>2500.0            </td><td>1815.0            </td><td>657.0             </td><td>1000.0            </td><td>1000.0            </td><td>800.0             </td><td>0                   </td></tr>\n",
       "<tr><td>6      </td><td>7.0              </td><td>500000.0          </td><td>male  </td><td>graduate school</td><td>single    </td><td>29.0             </td><td>use of revolving credit</td><td>use of revolving credit</td><td>use of revolving credit</td><td>use of revolving credit</td><td>use of revolving credit</td><td>use of revolving credit</td><td>367965.0         </td><td>412023.0         </td><td>445007.0         </td><td>542653.0         </td><td>483003.0         </td><td>473944.0         </td><td>55000.0           </td><td>40000.0           </td><td>38000.0           </td><td>20239.0           </td><td>13750.0           </td><td>13770.0           </td><td>0                   </td></tr>\n",
       "<tr><td>7      </td><td>8.0              </td><td>100000.0          </td><td>female</td><td>university     </td><td>single    </td><td>23.0             </td><td>use of revolving credit</td><td>pay duly               </td><td>pay duly               </td><td>use of revolving credit</td><td>use of revolving credit</td><td>pay duly               </td><td>11876.0          </td><td>380.0            </td><td>601.0            </td><td>221.0            </td><td>-159.0           </td><td>567.0            </td><td>380.0             </td><td>601.0             </td><td>0.0               </td><td>581.0             </td><td>1687.0            </td><td>1542.0            </td><td>0                   </td></tr>\n",
       "<tr><td>8      </td><td>9.0              </td><td>140000.0          </td><td>female</td><td>high school    </td><td>married   </td><td>28.0             </td><td>use of revolving credit</td><td>use of revolving credit</td><td>2 month delay          </td><td>use of revolving credit</td><td>use of revolving credit</td><td>use of revolving credit</td><td>11285.0          </td><td>14096.0          </td><td>12108.0          </td><td>12211.0          </td><td>11793.0          </td><td>3719.0           </td><td>3329.0            </td><td>0.0               </td><td>432.0             </td><td>1000.0            </td><td>1000.0            </td><td>1000.0            </td><td>0                   </td></tr>\n",
       "<tr><td>9      </td><td>10.0             </td><td>20000.0           </td><td>male  </td><td>high school    </td><td>single    </td><td>35.0             </td><td>no consumption         </td><td>no consumption         </td><td>no consumption         </td><td>no consumption         </td><td>pay duly               </td><td>pay duly               </td><td>0.0              </td><td>0.0              </td><td>0.0              </td><td>0.0              </td><td>13007.0          </td><td>13912.0          </td><td>0.0               </td><td>0.0               </td><td>0.0               </td><td>13007.0           </td><td>1122.0            </td><td>0.0               </td><td>0                   </td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Train an H2O GBM classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Split data into training and test sets for early stopping\n",
    "The credit card default data is split into training and test sets to monitor and prevent overtraining. Reproducibility is also an important factor in creating trustworthy models, and randomly splitting datasets can introduce randomness in model predictions and other results. A random seed is used here to ensure the data split is reproducible."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train data rows = 21060, columns = 25\n",
      "Test data rows = 8940, columns = 25\n"
     ]
    }
   ],
   "source": [
    "# split into training and validation\n",
    "train, test = data.split_frame([0.7], seed=12345)\n",
    "\n",
    "# summarize split\n",
    "print('Train data rows = %d, columns = %d' % (train.shape[0], train.shape[1]))\n",
    "print('Test data rows = %d, columns = %d' % (test.shape[0], test.shape[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train h2o GBM classifier\n",
    "Many tuning parameters must be specified to train a GBM using h2o. Typically a grid search would be performed to identify the best parameters for a given modeling task using the `H2OGridSearch` class. For brevity's sake, a previously-discovered set of good tuning parameters are specified here. Because gradient boosting methods typically resample training data, an additional random seed is also specified for the h2o GBM using the `seed` parameter to create reproducible predictions, error rates, and variable importance values. To avoid overfitting, the `stopping_rounds` parameter is used to stop the training process after the test error fails to decrease for 5 iterations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gbm Model Build progress: |███████████████████████████████████████████████| 100%\n",
      "GBM Test AUC = 0.78\n"
     ]
    }
   ],
   "source": [
    "# initialize GBM model\n",
    "model = H2OGradientBoostingEstimator(ntrees=150,            # maximum 150 trees in GBM\n",
    "                                     max_depth=4,           # trees can have maximum depth of 4\n",
    "                                     sample_rate=0.9,       # use 90% of rows in each iteration (tree)\n",
    "                                     col_sample_rate=0.9,   # use 90% of variables in each iteration (tree)\n",
    "                                     stopping_rounds=5,     # stop if validation error does not decrease for 5 iterations (trees)\n",
    "                                     score_tree_interval=1, # for reproducibility, set higher for bigger data\n",
    "                                     seed=12345)            # random seed for reproducibility\n",
    "\n",
    "# train a GBM model\n",
    "model.train(y=y, x=X, training_frame=train, validation_frame=test)\n",
    "\n",
    "# print AUC\n",
    "print('GBM Test AUC = %.2f' % model.auc(valid=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Use LIME to generate descriptions for a local region with a perturbed sample\n",
    "\n",
    "LIME was originally described in the context of explaining image or text classification decisions here: http://www.kdd.org/kdd2016/papers/files/rfp0573-ribeiroA.pdf. It can certainly also be applied to business or customer data, as will be done in the remaining sections of this notebook. Multiple Python implementations of LIME are available from the original authors of LIME, from the eli5 package, from the skater package, and probably others. However, this notebook uses a simple, step-by-step implementation of LIME for instructional purposes. \n",
    "\n",
    "A linear model cannot be built on a single observation, so LIME typically requires that a set of rows similar to the row of interest be simulated. This set of records are scored using the complex model to be explained. Then the records are weighted by their closeness to the record of interest, and a regularized linear model is trained on this weighted explanatory set. The parameters of the linear model and LIME-derived reason codes are then used to explain the prediction for the selected record. Because simulation of new points can seem abstract to some practicioners and simulation and distance calculations can be somewhat burdensome for creating explanations quickly in mission-critical applications, this notebook also presents a variation of LIME in which a more practical sample, instead of a perturbed, simulated sample, is used to create a local region in which to fit a linear model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display the most risky customer\n",
    "In the Oriole notebook *Increase Transparency and Accountability in Your Machine Learning Project with Python and H2O*, row index 29116 was found to contain the riskiest customer in the test dataset according to the h2o GBM model. Sections 3-7 focus on deriving reason codes and other explanations for this customer's GBM prediction. The riskiest customer is selected first for analysis as an exercise in boundary testing."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">   ID</th><th style=\"text-align: right;\">  LIMIT_BAL</th><th>SEX   </th><th>EDUCATION  </th><th>MARRIAGE  </th><th style=\"text-align: right;\">  AGE</th><th>PAY_0        </th><th>PAY_2        </th><th>PAY_3        </th><th>PAY_4        </th><th>PAY_5        </th><th>PAY_6        </th><th style=\"text-align: right;\">  BILL_AMT1</th><th style=\"text-align: right;\">  BILL_AMT2</th><th style=\"text-align: right;\">  BILL_AMT3</th><th style=\"text-align: right;\">  BILL_AMT4</th><th style=\"text-align: right;\">  BILL_AMT5</th><th style=\"text-align: right;\">  BILL_AMT6</th><th style=\"text-align: right;\">  PAY_AMT1</th><th style=\"text-align: right;\">  PAY_AMT2</th><th style=\"text-align: right;\">  PAY_AMT3</th><th style=\"text-align: right;\">  PAY_AMT4</th><th style=\"text-align: right;\">  PAY_AMT5</th><th style=\"text-align: right;\">  PAY_AMT6</th><th style=\"text-align: right;\">  DEFAULT_NEXT_MONTH</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">29116</td><td style=\"text-align: right;\">      20000</td><td>female</td><td>university </td><td>married   </td><td style=\"text-align: right;\">   59</td><td>3 month delay</td><td>2 month delay</td><td>3 month delay</td><td>2 month delay</td><td>2 month delay</td><td>4 month delay</td><td style=\"text-align: right;\">       8803</td><td style=\"text-align: right;\">      11137</td><td style=\"text-align: right;\">      10672</td><td style=\"text-align: right;\">      11201</td><td style=\"text-align: right;\">      12721</td><td style=\"text-align: right;\">      11946</td><td style=\"text-align: right;\">      2800</td><td style=\"text-align: right;\">         0</td><td style=\"text-align: right;\">      1000</td><td style=\"text-align: right;\">      2000</td><td style=\"text-align: right;\">         0</td><td style=\"text-align: right;\">         0</td><td style=\"text-align: right;\">                   1</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "row = test[test['ID'] == 29116]\n",
    "row"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use LIME, a sample of similar (i.e., near or local) points is simulated around the customer of interest. This simple function draws numeric values from normal distributions centered around the customer of interest and draws categorical values at random from the variable values in the test set."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>LIMIT_BAL</th>\n",
       "      <th>SEX</th>\n",
       "      <th>EDUCATION</th>\n",
       "      <th>MARRIAGE</th>\n",
       "      <th>AGE</th>\n",
       "      <th>PAY_0</th>\n",
       "      <th>PAY_2</th>\n",
       "      <th>PAY_3</th>\n",
       "      <th>PAY_4</th>\n",
       "      <th>PAY_5</th>\n",
       "      <th>...</th>\n",
       "      <th>BILL_AMT3</th>\n",
       "      <th>BILL_AMT4</th>\n",
       "      <th>BILL_AMT5</th>\n",
       "      <th>BILL_AMT6</th>\n",
       "      <th>PAY_AMT1</th>\n",
       "      <th>PAY_AMT2</th>\n",
       "      <th>PAY_AMT3</th>\n",
       "      <th>PAY_AMT4</th>\n",
       "      <th>PAY_AMT5</th>\n",
       "      <th>PAY_AMT6</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>9988.454213</td>\n",
       "      <td>female</td>\n",
       "      <td>graduate school</td>\n",
       "      <td>divorced</td>\n",
       "      <td>58.287510</td>\n",
       "      <td>5 month delay</td>\n",
       "      <td>5 month delay</td>\n",
       "      <td>5 month delay</td>\n",
       "      <td>5 month delay</td>\n",
       "      <td>6 month delay</td>\n",
       "      <td>...</td>\n",
       "      <td>5433.340804</td>\n",
       "      <td>6276.576876</td>\n",
       "      <td>8055.530587</td>\n",
       "      <td>7347.467911</td>\n",
       "      <td>1597.834490</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1000.000000</td>\n",
       "      <td>823.253257</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>181039.642122</td>\n",
       "      <td>male</td>\n",
       "      <td>high school</td>\n",
       "      <td>married</td>\n",
       "      <td>70.460689</td>\n",
       "      <td>pay duly</td>\n",
       "      <td>pay duly</td>\n",
       "      <td>pay duly</td>\n",
       "      <td>pay duly</td>\n",
       "      <td>use of revolving credit</td>\n",
       "      <td>...</td>\n",
       "      <td>94937.888614</td>\n",
       "      <td>90412.278099</td>\n",
       "      <td>87766.906051</td>\n",
       "      <td>85915.192926</td>\n",
       "      <td>22137.303918</td>\n",
       "      <td>25583.930273</td>\n",
       "      <td>21802.010398</td>\n",
       "      <td>20928.433066</td>\n",
       "      <td>19123.775929</td>\n",
       "      <td>22563.515833</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>20000.000000</td>\n",
       "      <td>male</td>\n",
       "      <td>university</td>\n",
       "      <td>single</td>\n",
       "      <td>43.284233</td>\n",
       "      <td>7 month delay</td>\n",
       "      <td>7 month delay</td>\n",
       "      <td>7 month delay</td>\n",
       "      <td>7 month delay</td>\n",
       "      <td>8 month delay</td>\n",
       "      <td>...</td>\n",
       "      <td>10672.000000</td>\n",
       "      <td>11201.000000</td>\n",
       "      <td>12721.000000</td>\n",
       "      <td>11946.000000</td>\n",
       "      <td>2800.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>1000.000000</td>\n",
       "      <td>2000.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       LIMIT_BAL     SEX        EDUCATION  MARRIAGE        AGE          PAY_0  \\\n",
       "0    9988.454213  female  graduate school  divorced  58.287510  5 month delay   \n",
       "1  181039.642122    male      high school   married  70.460689       pay duly   \n",
       "2   20000.000000    male       university    single  43.284233  7 month delay   \n",
       "\n",
       "           PAY_2          PAY_3          PAY_4                    PAY_5  \\\n",
       "0  5 month delay  5 month delay  5 month delay            6 month delay   \n",
       "1       pay duly       pay duly       pay duly  use of revolving credit   \n",
       "2  7 month delay  7 month delay  7 month delay            8 month delay   \n",
       "\n",
       "       ...          BILL_AMT3     BILL_AMT4     BILL_AMT5     BILL_AMT6  \\\n",
       "0      ...        5433.340804   6276.576876   8055.530587   7347.467911   \n",
       "1      ...       94937.888614  90412.278099  87766.906051  85915.192926   \n",
       "2      ...       10672.000000  11201.000000  12721.000000  11946.000000   \n",
       "\n",
       "       PAY_AMT1      PAY_AMT2      PAY_AMT3      PAY_AMT4      PAY_AMT5  \\\n",
       "0   1597.834490      0.000000   1000.000000    823.253257      0.000000   \n",
       "1  22137.303918  25583.930273  21802.010398  20928.433066  19123.775929   \n",
       "2   2800.000000      0.000000   1000.000000   2000.000000      0.000000   \n",
       "\n",
       "       PAY_AMT6  \n",
       "0      0.000000  \n",
       "1  22563.515833  \n",
       "2      0.000000  \n",
       "\n",
       "[3 rows x 23 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def generate_local_sample(row, frame, X, N=1000):\n",
    "    \n",
    "    \"\"\" Generates a perturbed sample around a row of interest.\n",
    "    \n",
    "    Args:\n",
    "        row: Row of H2OFrame to be explained.\n",
    "        frame: H2OFrame in which row is stored.\n",
    "        X: List of model input variables.\n",
    "        N: Number of samples to generate.\n",
    "    \n",
    "    Returns:\n",
    "        Pandas DataFrame containing perturbed sample.\n",
    "    \n",
    "    \"\"\"\n",
    "    \n",
    "    # initialize Pandas DataFrame\n",
    "    sample_frame = pd.DataFrame(data=np.zeros(shape=(N, len(X))), columns=X)\n",
    "    \n",
    "    # generate column vectors of \n",
    "    # randomly drawn levels for categorical variables\n",
    "    # normally distributed numeric values around mean of column for numeric variables\n",
    "    for key, val in frame[X].types.items():\n",
    "        if val == 'enum': # 'enum' means categorical\n",
    "            rs = np.random.RandomState(11111) # random seed for reproducibility\n",
    "            draw = rs.choice(frame[key].levels()[0], size=(1, N))[0]\n",
    "        else:\n",
    "            rs = np.random.RandomState(11111) # random seed for reproducibility\n",
    "            loc = row[key][0, 0]\n",
    "            sd = frame[key].sd()\n",
    "            draw = rs.normal(loc, sd, (N, 1))\n",
    "            draw[draw < 0] = loc # prevents unrealistic values when std. dev. is large\n",
    "        \n",
    "        sample_frame[key] = draw\n",
    "        \n",
    "    return sample_frame\n",
    "\n",
    "# run and display results\n",
    "perturbed_sample = generate_local_sample(row, test, X)\n",
    "perturbed_sample.head(n=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Calculate distance between row of interest and perturbed sample\n",
    "Once the sample is simulated, then distances from the point of interest are used to weigh each point before fitting a penalized regression model. Since Euclidean distance calculations require numeric quanitites, categorical input variables are one-hot encoded. (Pandas has convenient functionality for one-hot encoding, and the H2OFrames are temporarily casted back to Pandas DataFrames to perform the encoding.) To prevent the disparate scales of numeric values, such as `AGE` and `LIMIT_BAL`, from  skewing Euclidean distances, numeric input variables are standardized. \n",
    "\n",
    "First, the row containing the riskiest customer is encoded and standardized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  LIMIT_BAL</th><th style=\"text-align: right;\">     AGE</th><th style=\"text-align: right;\">  BILL_AMT1</th><th style=\"text-align: right;\">  BILL_AMT2</th><th style=\"text-align: right;\">  BILL_AMT3</th><th style=\"text-align: right;\">  BILL_AMT4</th><th style=\"text-align: right;\">  BILL_AMT5</th><th style=\"text-align: right;\">  BILL_AMT6</th><th style=\"text-align: right;\">  PAY_AMT1</th><th style=\"text-align: right;\">  PAY_AMT2</th><th style=\"text-align: right;\">  PAY_AMT3</th><th style=\"text-align: right;\">  PAY_AMT4</th><th style=\"text-align: right;\">  PAY_AMT5</th><th style=\"text-align: right;\">  PAY_AMT6</th><th style=\"text-align: right;\">  SEX_female</th><th style=\"text-align: right;\">  EDUCATION_graduate school</th><th style=\"text-align: right;\">  MARRIAGE_married</th><th style=\"text-align: right;\">  PAY_0_3 month delay</th><th style=\"text-align: right;\">  PAY_2_2 month delay</th><th style=\"text-align: right;\">  PAY_3_2 month delay</th><th style=\"text-align: right;\">  PAY_4_3 month delay</th><th style=\"text-align: right;\">  PAY_5_3 month delay</th><th style=\"text-align: right;\">  PAY_6_3 month delay</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">    2.24639</td><td style=\"text-align: right;\">0.481433</td><td style=\"text-align: right;\">   -0.66112</td><td style=\"text-align: right;\">  -0.657958</td><td style=\"text-align: right;\">  -0.651883</td><td style=\"text-align: right;\">  -0.637776</td><td style=\"text-align: right;\">  -0.622867</td><td style=\"text-align: right;\">  -0.609179</td><td style=\"text-align: right;\"> -0.360791</td><td style=\"text-align: right;\"> -0.282325</td><td style=\"text-align: right;\"> -0.315203</td><td style=\"text-align: right;\"> -0.319038</td><td style=\"text-align: right;\"> -0.319074</td><td style=\"text-align: right;\"> -0.270536</td><td style=\"text-align: right;\">           1</td><td style=\"text-align: right;\">                          1</td><td style=\"text-align: right;\">                 1</td><td style=\"text-align: right;\">                    1</td><td style=\"text-align: right;\">                    1</td><td style=\"text-align: right;\">                    1</td><td style=\"text-align: right;\">                    1</td><td style=\"text-align: right;\">                    1</td><td style=\"text-align: right;\">                    1</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# scaling and one-hot encoding for calculating Euclidian distance\n",
    "# for the row of interest\n",
    "\n",
    "# scale numeric\n",
    "numeric = list(set(X) - set(['ID', 'SEX', 'EDUCATION', 'MARRIAGE', 'PAY_0', 'PAY_2',\n",
    "                             'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6', 'DEFAULT_NEXT_MONTH']))\n",
    "\n",
    "scaled_test = test.as_data_frame()\n",
    "scaled_test[numeric] = (scaled_test[numeric] - scaled_test[numeric].mean())/scaled_test[numeric].std()\n",
    "    \n",
    "# encode categorical\n",
    "row_df = scaled_test[scaled_test['ID'] == 22760]\n",
    "row_dummies = pd.concat([row_df.drop(['ID', 'SEX', 'EDUCATION', 'MARRIAGE', 'PAY_0', 'PAY_2',\n",
    "                                      'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6', 'DEFAULT_NEXT_MONTH'], axis=1),\n",
    "                        pd.get_dummies(row_df[['SEX', 'EDUCATION', 'MARRIAGE', 'PAY_0',\n",
    "                                               'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6']])], \n",
    "                        axis=1)\n",
    "\n",
    "# convert to H2OFrame\n",
    "row_dummies = h2o.H2OFrame(row_dummies)\n",
    "row_dummies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then the simulated sample is encoded and standardized."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  LIMIT_BAL</th><th style=\"text-align: right;\">       AGE</th><th style=\"text-align: right;\">  BILL_AMT1</th><th style=\"text-align: right;\">  BILL_AMT2</th><th style=\"text-align: right;\">  BILL_AMT3</th><th style=\"text-align: right;\">  BILL_AMT4</th><th style=\"text-align: right;\">  BILL_AMT5</th><th style=\"text-align: right;\">  BILL_AMT6</th><th style=\"text-align: right;\">  PAY_AMT1</th><th style=\"text-align: right;\">  PAY_AMT2</th><th style=\"text-align: right;\">  PAY_AMT3</th><th style=\"text-align: right;\">  PAY_AMT4</th><th style=\"text-align: right;\">  PAY_AMT5</th><th style=\"text-align: right;\">  PAY_AMT6</th><th style=\"text-align: right;\">  SEX_female</th><th style=\"text-align: right;\">  EDUCATION_graduate school</th><th style=\"text-align: right;\">  MARRIAGE_married</th><th style=\"text-align: right;\">  PAY_0_3 month delay</th><th style=\"text-align: right;\">  PAY_2_2 month delay</th><th style=\"text-align: right;\">  PAY_3_2 month delay</th><th style=\"text-align: right;\">  PAY_4_3 month delay</th><th style=\"text-align: right;\">  PAY_5_3 month delay</th><th style=\"text-align: right;\">  PAY_6_3 month delay</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">  -0.845634</td><td style=\"text-align: right;\">-0.0955699</td><td style=\"text-align: right;\">  -0.84979 </td><td style=\"text-align: right;\">  -0.845174</td><td style=\"text-align: right;\">  -0.845174</td><td style=\"text-align: right;\">  -0.83886 </td><td style=\"text-align: right;\">  -0.834367</td><td style=\"text-align: right;\">  -0.834967</td><td style=\"text-align: right;\"> -0.83834 </td><td style=\"text-align: right;\"> -0.723503</td><td style=\"text-align: right;\"> -0.722209</td><td style=\"text-align: right;\"> -0.849042</td><td style=\"text-align: right;\"> -0.723503</td><td style=\"text-align: right;\"> -0.723503</td><td style=\"text-align: right;\">           1</td><td style=\"text-align: right;\">                          1</td><td style=\"text-align: right;\">                 0</td><td style=\"text-align: right;\">                    0</td><td style=\"text-align: right;\">                    0</td><td style=\"text-align: right;\">                    0</td><td style=\"text-align: right;\">                    0</td><td style=\"text-align: right;\">                    0</td><td style=\"text-align: right;\">                    0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">   1.41903 </td><td style=\"text-align: right;\"> 1.2011   </td><td style=\"text-align: right;\">   1.41949 </td><td style=\"text-align: right;\">   1.41897 </td><td style=\"text-align: right;\">   1.41897 </td><td style=\"text-align: right;\">   1.4182  </td><td style=\"text-align: right;\">   1.4176  </td><td style=\"text-align: right;\">   1.41769 </td><td style=\"text-align: right;\">  1.41814 </td><td style=\"text-align: right;\">  1.41999 </td><td style=\"text-align: right;\">  1.41993 </td><td style=\"text-align: right;\">  1.41941 </td><td style=\"text-align: right;\">  1.41999 </td><td style=\"text-align: right;\">  1.41999 </td><td style=\"text-align: right;\">           0</td><td style=\"text-align: right;\">                          0</td><td style=\"text-align: right;\">                 1</td><td style=\"text-align: right;\">                    0</td><td style=\"text-align: right;\">                    0</td><td style=\"text-align: right;\">                    0</td><td style=\"text-align: right;\">                    0</td><td style=\"text-align: right;\">                    0</td><td style=\"text-align: right;\">                    0</td></tr>\n",
       "<tr><td style=\"text-align: right;\">  -0.713084</td><td style=\"text-align: right;\">-1.69369  </td><td style=\"text-align: right;\">  -0.716971</td><td style=\"text-align: right;\">  -0.712654</td><td style=\"text-align: right;\">  -0.712654</td><td style=\"text-align: right;\">  -0.706755</td><td style=\"text-align: right;\">  -0.70256 </td><td style=\"text-align: right;\">  -0.70312 </td><td style=\"text-align: right;\"> -0.706269</td><td style=\"text-align: right;\"> -0.723503</td><td style=\"text-align: right;\"> -0.722209</td><td style=\"text-align: right;\"> -0.716271</td><td style=\"text-align: right;\"> -0.723503</td><td style=\"text-align: right;\"> -0.723503</td><td style=\"text-align: right;\">           0</td><td style=\"text-align: right;\">                          0</td><td style=\"text-align: right;\">                 0</td><td style=\"text-align: right;\">                    0</td><td style=\"text-align: right;\">                    0</td><td style=\"text-align: right;\">                    0</td><td style=\"text-align: right;\">                    0</td><td style=\"text-align: right;\">                    0</td><td style=\"text-align: right;\">                    0</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# scaling and one-hot encoding for calculating Euclidian distance \n",
    "# for the simulated sample\n",
    "\n",
    "# scale\n",
    "scaled_perturbed_sample = perturbed_sample[numeric].copy(deep=True)\n",
    "scaled_perturbed_sample = (scaled_perturbed_sample - scaled_perturbed_sample.mean())/scaled_perturbed_sample.std()\n",
    "\n",
    "# encode\n",
    "perturbed_sample_dummies = pd.concat([scaled_perturbed_sample,\n",
    "                                      pd.get_dummies(perturbed_sample[['SEX', 'EDUCATION', 'MARRIAGE', 'PAY_0',\n",
    "                                                                       'PAY_2', 'PAY_3', 'PAY_4', 'PAY_5', 'PAY_6']])],\n",
    "                                     axis=1)\n",
    "\n",
    "# convert to H2OFrame\n",
    "perturbed_sample_dummies = h2o.H2OFrame(perturbed_sample_dummies[row_dummies.columns])\n",
    "perturbed_sample_dummies.head(rows=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Distance is calculated using h2o. The distance is substracted from the maximum distance, changing the distance values into similarity values. Now the observations with the highest values are those that are closest to the observation of interest and they will carry the most weight in the local explanatory linear model. A few sample similarity values are displayed directly below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  distance</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">   13.2143</td></tr>\n",
       "<tr><td style=\"text-align: right;\">   10.2477</td></tr>\n",
       "<tr><td style=\"text-align: right;\">   12.6506</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate distance using H2OFrame distance function\n",
    "distance = row_dummies.distance(perturbed_sample_dummies, measure='l2').transpose()\n",
    "distance.columns = ['distance']          # rename \n",
    "distance = distance.max() - distance     # lower distances, higher weight in LIME\n",
    "distance.head(rows=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Bind distance weights onto perturbed sample\n",
    "To fit an h2o linear model using the similarities as observation weights, the distance column must reside in the same H2OFrame as the simulated sample data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Parse progress: |█████████████████████████████████████████████████████████| 100%\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  LIMIT_BAL</th><th>SEX   </th><th>EDUCATION      </th><th>MARRIAGE  </th><th style=\"text-align: right;\">    AGE</th><th>PAY_0        </th><th>PAY_2        </th><th>PAY_3        </th><th>PAY_4        </th><th>PAY_5                  </th><th>PAY_6                  </th><th style=\"text-align: right;\">  BILL_AMT1</th><th style=\"text-align: right;\">  BILL_AMT2</th><th style=\"text-align: right;\">  BILL_AMT3</th><th style=\"text-align: right;\">  BILL_AMT4</th><th style=\"text-align: right;\">  BILL_AMT5</th><th style=\"text-align: right;\">  BILL_AMT6</th><th style=\"text-align: right;\">  PAY_AMT1</th><th style=\"text-align: right;\">  PAY_AMT2</th><th style=\"text-align: right;\">  PAY_AMT3</th><th style=\"text-align: right;\">  PAY_AMT4</th><th style=\"text-align: right;\">  PAY_AMT5</th><th style=\"text-align: right;\">  PAY_AMT6</th><th style=\"text-align: right;\">  distance</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">    9988.45</td><td>female</td><td>graduate school</td><td>divorced  </td><td style=\"text-align: right;\">58.2875</td><td>5 month delay</td><td>5 month delay</td><td>5 month delay</td><td>5 month delay</td><td>6 month delay          </td><td>6 month delay          </td><td style=\"text-align: right;\">    3152.62</td><td style=\"text-align: right;\">    5678.04</td><td style=\"text-align: right;\">    5433.34</td><td style=\"text-align: right;\">    6276.58</td><td style=\"text-align: right;\">    8055.53</td><td style=\"text-align: right;\">    7347.47</td><td style=\"text-align: right;\">   1597.83</td><td style=\"text-align: right;\">       0  </td><td style=\"text-align: right;\">      1000</td><td style=\"text-align: right;\">   823.253</td><td style=\"text-align: right;\">       0  </td><td style=\"text-align: right;\">       0  </td><td style=\"text-align: right;\">   13.2143</td></tr>\n",
       "<tr><td style=\"text-align: right;\">  181040   </td><td>male  </td><td>high school    </td><td>married   </td><td style=\"text-align: right;\">70.4607</td><td>pay duly     </td><td>pay duly     </td><td>pay duly     </td><td>pay duly     </td><td>use of revolving credit</td><td>use of revolving credit</td><td style=\"text-align: right;\">   99691.6 </td><td style=\"text-align: right;\">   98946.5 </td><td style=\"text-align: right;\">   94937.9 </td><td style=\"text-align: right;\">   90412.3 </td><td style=\"text-align: right;\">   87766.9 </td><td style=\"text-align: right;\">   85915.2 </td><td style=\"text-align: right;\">  22137.3 </td><td style=\"text-align: right;\">   25583.9</td><td style=\"text-align: right;\">     21802</td><td style=\"text-align: right;\"> 20928.4  </td><td style=\"text-align: right;\">   19123.8</td><td style=\"text-align: right;\">   22563.5</td><td style=\"text-align: right;\">   10.2477</td></tr>\n",
       "<tr><td style=\"text-align: right;\">   20000   </td><td>male  </td><td>university     </td><td>single    </td><td style=\"text-align: right;\">43.2842</td><td>7 month delay</td><td>7 month delay</td><td>7 month delay</td><td>7 month delay</td><td>8 month delay          </td><td>8 month delay          </td><td style=\"text-align: right;\">    8803   </td><td style=\"text-align: right;\">   11137   </td><td style=\"text-align: right;\">   10672   </td><td style=\"text-align: right;\">   11201   </td><td style=\"text-align: right;\">   12721   </td><td style=\"text-align: right;\">   11946   </td><td style=\"text-align: right;\">   2800   </td><td style=\"text-align: right;\">       0  </td><td style=\"text-align: right;\">      1000</td><td style=\"text-align: right;\">  2000    </td><td style=\"text-align: right;\">       0  </td><td style=\"text-align: right;\">       0  </td><td style=\"text-align: right;\">   12.6506</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "perturbed_sample = h2o.H2OFrame(perturbed_sample).cbind(distance)\n",
    "perturbed_sample.head(rows=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Bind model predictions onto perturbed sample\n",
    "For LIME, the target of the explanatory local linear model is the predictions of the GBM model in the local simulated sample. The values are calculated and column-bound to the simulated sample. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gbm prediction progress: |████████████████████████████████████████████████| 100%\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  LIMIT_BAL</th><th>SEX   </th><th>EDUCATION      </th><th>MARRIAGE  </th><th style=\"text-align: right;\">    AGE</th><th>PAY_0        </th><th>PAY_2        </th><th>PAY_3        </th><th>PAY_4        </th><th>PAY_5                  </th><th>PAY_6                  </th><th style=\"text-align: right;\">  BILL_AMT1</th><th style=\"text-align: right;\">  BILL_AMT2</th><th style=\"text-align: right;\">  BILL_AMT3</th><th style=\"text-align: right;\">  BILL_AMT4</th><th style=\"text-align: right;\">  BILL_AMT5</th><th style=\"text-align: right;\">  BILL_AMT6</th><th style=\"text-align: right;\">  PAY_AMT1</th><th style=\"text-align: right;\">  PAY_AMT2</th><th style=\"text-align: right;\">  PAY_AMT3</th><th style=\"text-align: right;\">  PAY_AMT4</th><th style=\"text-align: right;\">  PAY_AMT5</th><th style=\"text-align: right;\">  PAY_AMT6</th><th style=\"text-align: right;\">  distance</th><th style=\"text-align: right;\">  p_DEFAULT_NEXT_MONTH</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">    9988.45</td><td>female</td><td>graduate school</td><td>divorced  </td><td style=\"text-align: right;\">58.2875</td><td>5 month delay</td><td>5 month delay</td><td>5 month delay</td><td>5 month delay</td><td>6 month delay          </td><td>6 month delay          </td><td style=\"text-align: right;\">    3152.62</td><td style=\"text-align: right;\">    5678.04</td><td style=\"text-align: right;\">    5433.34</td><td style=\"text-align: right;\">    6276.58</td><td style=\"text-align: right;\">    8055.53</td><td style=\"text-align: right;\">    7347.47</td><td style=\"text-align: right;\">   1597.83</td><td style=\"text-align: right;\">       0  </td><td style=\"text-align: right;\">      1000</td><td style=\"text-align: right;\">   823.253</td><td style=\"text-align: right;\">       0  </td><td style=\"text-align: right;\">       0  </td><td style=\"text-align: right;\">   13.2143</td><td style=\"text-align: right;\">             0.515197 </td></tr>\n",
       "<tr><td style=\"text-align: right;\">  181040   </td><td>male  </td><td>high school    </td><td>married   </td><td style=\"text-align: right;\">70.4607</td><td>pay duly     </td><td>pay duly     </td><td>pay duly     </td><td>pay duly     </td><td>use of revolving credit</td><td>use of revolving credit</td><td style=\"text-align: right;\">   99691.6 </td><td style=\"text-align: right;\">   98946.5 </td><td style=\"text-align: right;\">   94937.9 </td><td style=\"text-align: right;\">   90412.3 </td><td style=\"text-align: right;\">   87766.9 </td><td style=\"text-align: right;\">   85915.2 </td><td style=\"text-align: right;\">  22137.3 </td><td style=\"text-align: right;\">   25583.9</td><td style=\"text-align: right;\">     21802</td><td style=\"text-align: right;\"> 20928.4  </td><td style=\"text-align: right;\">   19123.8</td><td style=\"text-align: right;\">   22563.5</td><td style=\"text-align: right;\">   10.2477</td><td style=\"text-align: right;\">             0.0663977</td></tr>\n",
       "<tr><td style=\"text-align: right;\">   20000   </td><td>male  </td><td>university     </td><td>single    </td><td style=\"text-align: right;\">43.2842</td><td>7 month delay</td><td>7 month delay</td><td>7 month delay</td><td>7 month delay</td><td>8 month delay          </td><td>8 month delay          </td><td style=\"text-align: right;\">    8803   </td><td style=\"text-align: right;\">   11137   </td><td style=\"text-align: right;\">   10672   </td><td style=\"text-align: right;\">   11201   </td><td style=\"text-align: right;\">   12721   </td><td style=\"text-align: right;\">   11946   </td><td style=\"text-align: right;\">   2800   </td><td style=\"text-align: right;\">       0  </td><td style=\"text-align: right;\">      1000</td><td style=\"text-align: right;\">  2000    </td><td style=\"text-align: right;\">       0  </td><td style=\"text-align: right;\">       0  </td><td style=\"text-align: right;\">   12.6506</td><td style=\"text-align: right;\">             0.692681 </td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "yhat = 'p_DEFAULT_NEXT_MONTH'\n",
    "preds1 = model.predict(perturbed_sample).drop(['predict', 'p0'])\n",
    "preds1.columns = [yhat]\n",
    "perturbed_sample = perturbed_sample.cbind(preds1)\n",
    "perturbed_sample.head(rows=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train penalized linear model in local region \n",
    "Once the simulated sample has been weighted with distances and contains the GBM model predictions, a linear model is fit to the original inputs and the GBM model predictions, weighted by similarity to the row of interest. The trained GLM coefficients are helpful for understanding the local region of response function around the riskiest customer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "glm Model Build progress: |███████████████████████████████████████████████| 100%\n",
      "\n",
      "Local Positive GLM Coefficients:\n",
      "MARRIAGE.married:         0.005757170307944079\n",
      "EDUCATION.high school:    0.005757170307945231\n",
      "PAY_0.8 month delay:      0.0065111741471265\n",
      "PAY_4.8 month delay:      0.006511174147127027\n",
      "PAY_3.8 month delay:      0.006511174147127041\n",
      "PAY_2.8 month delay:      0.006511174147127305\n",
      "PAY_0.4 month delay:      0.010754792563841487\n",
      "PAY_2.4 month delay:      0.010754792563842083\n",
      "PAY_4.4 month delay:      0.010754792563842167\n",
      "PAY_3.4 month delay:      0.010754792563842194\n",
      "PAY_5.6 month delay:      0.013679631302936743\n",
      "PAY_6.6 month delay:      0.01367963130293677\n",
      "PAY_2.2 month delay:      0.02636889214079144\n",
      "PAY_0.2 month delay:      0.02636889214079166\n",
      "PAY_3.2 month delay:      0.02636889214079169\n",
      "PAY_4.2 month delay:      0.026368892140792216\n",
      "PAY_0.3 month delay:      0.030938016884379638\n",
      "PAY_2.3 month delay:      0.03093801688437975\n",
      "PAY_4.3 month delay:      0.030938016884379915\n",
      "PAY_3.3 month delay:      0.030938016884380054\n",
      "PAY_2.7 month delay:      0.03649976537701373\n",
      "PAY_3.7 month delay:      0.036499765377013785\n",
      "PAY_0.7 month delay:      0.03649976537701445\n",
      "PAY_4.7 month delay:      0.036499765377014534\n",
      "PAY_6.3 month delay:      0.04135072490974577\n",
      "PAY_5.3 month delay:      0.0413507249097461\n",
      "Intercept:                0.5259242681066165\n",
      "\n",
      "Local GLM R-square:\n",
      "0.84\n"
     ]
    }
   ],
   "source": [
    "# initialize\n",
    "local_glm1 = H2OGeneralizedLinearEstimator(weights_column='distance',\n",
    "                                           seed=12345)\n",
    "# train \n",
    "local_glm1.train(x=X, y=yhat, training_frame=perturbed_sample)\n",
    "\n",
    "# coefs\n",
    "print('\\nLocal Positive GLM Coefficients:')\n",
    "for c_name, c_val in sorted(local_glm1.coef().items(), key=operator.itemgetter(1)):\n",
    "    if c_val > 0.0:\n",
    "        print('%s %s' % (str(c_name + ':').ljust(25), c_val))\n",
    "        \n",
    "# r2\n",
    "print('\\nLocal GLM R-square:\\n%.2f' % local_glm1.r2())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The coefficients of the local linear model describe the average behavior of the GBM response function around the riskiest customer. In this local region, customers who missed payments are treated as the most likely to default."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Generate reason codes with LIME based on a perturbed sample\n",
    "This basic function uses the coefficients of the local linear explanatory model and the values in the row of interest to plot reason code values in a bar chart. The local GLM coefficient multiplied by the value in a specific row are estimates of how much each variable contributed to each prediction decision. These values can tell you how a variable and its values were weighted in any given decision by the model. These values are crucially important for machine learning interpretability and are often to referred to \"local feature importance\", \"reason codes\", or \"turn-down codes.\" The latter phrases are borrowed from credit scoring. Credit lenders in the U.S. must provide reasons for turning down certain credit applications in an automated fashion. Reason codes can be easily extracted from LIME local feature importance values by simply ranking the variables that played the largest role in any given decision."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def plot_local_contrib(row, model, X, g_pred=None, scale=False): \n",
    "\n",
    "    \"\"\" Plots reason codes in a bar chart. \n",
    "    \n",
    "    Args:\n",
    "    \n",
    "        row: Row of H2OFrame to be explained.\n",
    "        model: H2O linear model used for generating reason codes.\n",
    "        X: List of model input variables.\n",
    "        g_pred: Prediction of model to be explained, sometimes denoted g, used for scaling.\n",
    "        scale: Whether to rescale contributions to sum to model predictions.\n",
    "    \n",
    "    \"\"\"\n",
    "    \n",
    "    # initialize Pandas DataFrame to store results\n",
    "    local_contrib_frame = pd.DataFrame(columns=['Name', 'Local Contribution', 'Sign'])\n",
    "    \n",
    "    # multiply values in row by local glm coefficients    \n",
    "    for key, val in sorted(row[X].types.items()):\n",
    "        contrib = 0\n",
    "        name = ''\n",
    "        if val == 'enum':\n",
    "                level = row[key][0, 0]\n",
    "                name = '.'.join([str(key), str(level)])\n",
    "                if name in model.coef():\n",
    "                    contrib = model.coef()[name]\n",
    "        else:\n",
    "            name = key\n",
    "            if name in model.coef():\n",
    "                contrib = row[name][0, 0]*model.coef()[name]\n",
    "        \n",
    "        # save only non-zero values\n",
    "        if contrib != 0.0:\n",
    "            local_contrib_frame = local_contrib_frame.append({'Name': name,\n",
    "                                                              'Local Contribution': contrib,\n",
    "                                                              'Sign': contrib > 0}, \n",
    "                                                             ignore_index=True) \n",
    "\n",
    "    if scale:\n",
    "        scaler = (g_pred - model.coef()['Intercept']) /\\\n",
    "            local_contrib_frame['Local Contribution'].sum()\n",
    "        local_contrib_frame['Local Contribution']  *= scaler\n",
    "        \n",
    "    # plot\n",
    "    _ = local_contrib_frame.plot(x='Name',\n",
    "                                 y='Local Contribution',\n",
    "                                 kind='bar', \n",
    "                                 title='Reason Codes', \n",
    "                                 color=local_contrib_frame.Sign.map({True:'b', False:'g'}), \n",
    "                                 legend=False) \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Display reason codes\n",
    "Here it can be seen that the riskiest customer's prediction is driven by her values for payment variables. Specifically, the top five LIME-derived reason codes contributing to her high probability of default are:\n",
    "\n",
    "1. Most recent payment is 3 months delayed.\n",
    "2. 3rd most recent payment is 3 months delayed.\n",
    "3. 2nd most recent payment is 2 months delayed.\n",
    "4. 4th most recent payment is 2 months delayed.\n",
    "5. Customer is married.\n",
    "\n",
    "(Of course variables like `MARRIAGE`, `AGE`, and `SEX`should not be used in credit lending decisions. For a slightly more careful treatment of GBM in the context of fair lending see: https://github.com/jphall663/interpretable_machine_learning_with_python/blob/master/dia.ipynb)\n",
    "\n",
    "This result is somewhat aligned with LOCO-derived reason codes found in section 5 of the *Increase Transparency and Accountability in Your Machine Learning Project with Python and H2O* Oriole notebook. Both perspectives weigh the riskiest customer's most recent and 3rd most recent payments very heavily in the model's prediction. A minor discrepancy between LOCO- and LIME-derived reason codes is somewhat expected. LIME explanations are linear, do not consider interactions, and represent offsets from the local linear model intercept. LOCO importance values are nonlinear, do consider interactions, and do not explicitly consider a linear intercept or offset. **Because most currently-available explanatory techniques are approximate, it is recommended that users employ several different explanatory techniques and trust only consisent results across techniques. Also, as of h2o 3.24, Shapley values are supported for h2o GBM. Use Shapley values along with or instead of LIME for any high-stakes application.**\n",
    "\n",
    "It is also imperative to compare these results to domain knowledge and reasonable expectations. In this case, the LIME reason codes and linear model coefficients tell a relatively parsimonious story about the GBM's prediction behavior. If this was not so, steps should be taken to either reconcile or remove inconsistencies and unreasonable predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAF2CAYAAABu5+/BAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XmcXXV9//HXm4RNlmBDEE2AIEEtiIqOCEqrwg+EKoSyCLiASsWqVOrWgq0blipu1AWxWECgKigWjUVFEcSVZQJoQMWGAEJAZRMQFIi8f3+c75CbcbYkc7/3ZO77+XjcR+4958w973tg7mfOOd9FtomIiFir1wEiIqIdUhAiIgJIQYiIiCIFISIigBSEiIgoUhAiIgJIQYhYo0j6rqS/63WOmJpSEKLVJN0o6Q+Sfi/p15I+K2nDXudaGZJ2kvR1Sb+TdJekyyW9ute5IoZLQYg1wT62NwSeAewIHNvjPBMmaRfgIuASYB4wE3g9sHcvc0WMJAUh1hi2fw1cQFMYAJC0rqQPS/qVpN9I+rSk9cu6x0r6X0m3S7q7PJ/T8bOvkrRE0n2SbpD08rJ8LUn/KukmSb+VdKakGWXdXEmWdHjZ5x2S/mWM2B8CzrB9gu073Fho+6UdOV4raXE5e1gg6Qkd6/aQ9AtJ90j6JKDON5f0Gkk/L5/vAklbleWSdGLJf6+kRZKeuhqHP/pACkKsMcqX+d7A4o7FHwCeRFMk5gGzgXeVdWsBpwNbAVsCfwA+Wd5rA+DjwN62NwKeC1xdfu5V5fFC4InAhkM/12FX4MnA7sC7JP3lCHkfA+wCnDvGZ9oNeD/wUuDxwE3A2WXdpsD/AP8KbApcDzyv42fnA+8A9gdmAd8HvlBW7wn8dTk2M8r73zlajggAbOeRR2sfwI3A74H7AAPfATYp6wTcD2zTsf0uwA2jvNczgLvL8w2A3wEHAOsP2+47wBs6Xj8ZeBiYDswtOeZ0rL8cOGSE/c0u2z5ljM93KvDBjtcbln3NBQ4DLu1YJ+AW4O/K628AR3SsXwt4gKYA7gb8EtgZWKvX/x3zWDMeOUOINcF+bv6KfwHwFJq/lqH5q/gxwMJyw/Z3wDfLciQ9RtJ/lks/9wLfAzaRNM32/cDBwN8Dt0k6X9JTyvs+geYv9SE30RSDx3Us+3XH8wdovsiHuxt4hOYv/9GssC/bv6f5S352WXdzxzp3vqb54v9Yx2e/i6ZozLZ9Ec1ZzUnAbyWdImnjMXJEpCDEmsP2JcBngQ+XRXfQXAba3vYm5THDzQ1ogLfS/HX/HNsb01xCgXId3vYFtveg+cL+BfCZsv5Wmi/bIVsCy4DfrGTeB4Af05yFjGaFfZVLWTOBpcBtwBYd69T5mqY4vK7js29ie33bPyr7/7jtZwHb0Vw6evvK5I/+k4IQa5r/APaQ9HTbj9B8iZ8oaTMASbMlvahsuxFNwfidpL8A3j30JpIeJ2l++QJ+kOay1CNl9ReAN0vaujRx/XfgHNvLViHvPwGvkvR2STPLvp8u6eyOfb1a0jMkrVv2dZntG4Hzge0l7S9pOvAmYPOO9/40cKyk7cv7zpB0UHn+bEnPkbQ2zWW1P3Z8vogRpSDEGsX27cCZLL9x/M80N5kvLZeFLqQ5K4CmeKxPcyZxKc3lpCFrAW+h+Qv9LuD5NM1BAU4DzqK5xHQDzZfpP6xi3h/RXM/fDVgi6S7gFODrZf2FwDuBL9OcEWwDHFLW3QEcRHPj/E5gW+CHHe99HnACcHb57NewvDnrxjTF8m6aS1J30rR4ihiVmsuSERHR73KGEBERQApCREQUKQgREQGkIERERJGCEBERQNP7co2x6aabeu7cub2OERGxRlm4cOEdtmeNt90aVRDmzp3L4OBgr2NERKxRJN00/la5ZBQREUUKQkREACkIERFRpCBERASQghAREUUKQkREACkIERFRrFH9ECLWVNLqv8dUGak+x6K9coYQERFACkJERBQTKgiS9pJ0naTFko4ZYf26ks4p6y+TNLcs30nS1eXxE0l/O9H3jIiIusYtCJKmASfRzNW6HXCopO2GbXYEcLftecCJNPO8QjPH64DtZwB7Af8pafoE3zMiIiqayBnCTsBi20tsPwScDcwfts184Izy/Fxgd0my/YDtZWX5esDQraCJvGdERFQ0kYIwG7i54/UtZdmI25QCcA8wE0DScyRdCywC/r6sn8h7Un7+SEmDkgZvv/32CcSNiIhV0fWbyrYvs7098GzgWEnrreTPn2J7wPbArFnjDucdERGraCIFYSmwRcfrOWXZiNtImg7MAO7s3MD2z4HfA0+d4HtGRERFEykIVwDbStpa0jrAIcCCYdssAA4vzw8ELrLt8jPTASRtBTwFuHGC7xkRERWN21PZ9jJJRwEXANOA02xfK+k4YND2AuBU4CxJi4G7aL7gAXYFjpH0MPAI8AbbdwCM9J6T/NkiImIlyGtQH/CBgQFnCs1YE2W4huVyLOqTtND2wHjbpadyREQAKQgREVGkIEREBJCCEBERRQpCREQAKQgREVGkIEREBJCCEBERRQpCREQAKQgREVGkIEREBJCCEBERRQpCREQAKQgREVGkIEREBDCBCXIiYurIXATL5Vj8uZwhREQEkIIQERFFCkJERAApCBERUUyoIEjaS9J1khZLOmaE9etKOqesv0zS3LJ8D0kLJS0q/+7W8TPfLe95dXlsNlkfKiIiVt64rYwkTQNOAvYAbgGukLTA9s86NjsCuNv2PEmHACcABwN3APvYvlXSU4ELgNkdP/dy24OT9FkiImI1TOQMYSdgse0lth8CzgbmD9tmPnBGeX4usLsk2b7K9q1l+bXA+pLWnYzgERExuSZSEGYDN3e8voUV/8pfYRvby4B7gJnDtjkAuNL2gx3LTi+Xi94pTUar4IiIWFVVbipL2p7mMtLrOha/3PYOwF+VxytH+dkjJQ1KGrz99tu7HzYiok9NpCAsBbboeD2nLBtxG0nTgRnAneX1HOA84DDb1w/9gO2l5d/7gM/TXJr6M7ZPsT1ge2DWrFkT+UwREbEKJlIQrgC2lbS1pHWAQ4AFw7ZZABxenh8IXGTbkjYBzgeOsf3DoY0lTZe0aXm+NvAS4JrV+ygREbE6xi0I5Z7AUTQthH4OfNH2tZKOk7Rv2exUYKakxcBbgKGmqUcB84B3DWteui5wgaSfAlfTnGF8ZjI/WERErBx5DRqdaWBgwIODaaUaa562DKTWhhxtyNCmHDVIWmh7YLzt0lM5IiKAFISIiChSECIiAkhBiIiIIgUhIiKAFISIiChSECIiAkhBiIiIIgUhIiKAFISIiChSECIiAkhBiIiIIgUhIiKAFISIiChSECIiAkhBiIiIIgUhIiKAFISIiChSECIiAkhBiIiIIgUhIiKACRYESXtJuk7SYknHjLB+XUnnlPWXSZpblu8haaGkReXf3Tp+5lll+WJJH5ekyfpQERGx8sYtCJKmAScBewPbAYdK2m7YZkcAd9ueB5wInFCW3wHsY3sH4HDgrI6fORl4LbBteey1Gp8jIiJW00TOEHYCFtteYvsh4Gxg/rBt5gNnlOfnArtLku2rbN9all8LrF/OJh4PbGz7UtsGzgT2W+1PExERq2wiBWE2cHPH61vKshG3sb0MuAeYOWybA4ArbT9Ytr9lnPcEQNKRkgYlDd5+++0TiBsREauiyk1lSdvTXEZ63cr+rO1TbA/YHpg1a9bkh4uICGBiBWEpsEXH6zll2YjbSJoOzADuLK/nAOcBh9m+vmP7OeO8Z0REVDSRgnAFsK2krSWtAxwCLBi2zQKam8YABwIX2bakTYDzgWNs/3BoY9u3AfdK2rm0LjoM+OpqfpaIiFgN4xaEck/gKOAC4OfAF21fK+k4SfuWzU4FZkpaDLwFGGqaehQwD3iXpKvLY7Oy7g3AfwGLgeuBb0zWh4qIiJWnppHPmmFgYMCDg4O9jhGx0iajl81k/Kq2IUcbMrQpRw2SFtoeGG+79FSOiAggBSEiIooUhIiIAFIQIiKiSEGIiAggBSEiIooUhIiIAFIQIiKiSEGIiAggBSEiIooUhIiIAFIQIiKiSEGIiAggBSEiIooUhIiIAFIQIiKiSEGIiAggBSEiIooUhIiIAFIQIiKimFBBkLSXpOskLZZ0zAjr15V0Tll/maS5ZflMSRdL+r2kTw77me+W97y6PDabjA8UERGrZvp4G0iaBpwE7AHcAlwhaYHtn3VsdgRwt+15kg4BTgAOBv4IvBN4ankM93Lbg6v5GSIiYhJM5AxhJ2Cx7SW2HwLOBuYP22Y+cEZ5fi6wuyTZvt/2D2gKQ0REtNhECsJs4OaO17eUZSNuY3sZcA8wcwLvfXq5XPROSZrA9hER0SW9vKn8cts7AH9VHq8caSNJR0oalDR4++23Vw0YEdFPJlIQlgJbdLyeU5aNuI2k6cAM4M6x3tT20vLvfcDnaS5NjbTdKbYHbA/MmjVrAnEjImJVTKQgXAFsK2lrSesAhwALhm2zADi8PD8QuMi2R3tDSdMlbVqerw28BLhmZcNHRMTkGbeVke1lko4CLgCmAafZvlbSccCg7QXAqcBZkhYDd9EUDQAk3QhsDKwjaT9gT+Am4IJSDKYBFwKfmdRPFhERK0Vj/CHfOgMDAx4cTCvVWPNMRpOJyfhVbUOONmRoU44aJC20PTDedumpHBERQApCREQUKQgREQGkIERERJGCEBERQApCREQUKQgREQGkIERERJGCEBERQApCREQUKQgREQGkIERERJGCEBERQApCREQUKQgREQGkIERERJGCEBERQApCREQUKQgREQGkIERERJGCEBERwAQLgqS9JF0nabGkY0ZYv66kc8r6yyTNLctnSrpY0u8lfXLYzzxL0qLyMx+XpMn4QBERsWrGLQiSpgEnAXsD2wGHStpu2GZHAHfbngecCJxQlv8ReCfwthHe+mTgtcC25bHXqnyAiIiYHBM5Q9gJWGx7ie2HgLOB+cO2mQ+cUZ6fC+wuSbbvt/0DmsLwKEmPBza2faltA2cC+63OB4mIiNUzkYIwG7i54/UtZdmI29heBtwDzBznPW8Z5z0BkHSkpEFJg7fffvsE4kZExKpo/U1l26fYHrA9MGvWrF7HiYiYsiZSEJYCW3S8nlOWjbiNpOnADODOcd5zzjjvGRERFU2kIFwBbCtpa0nrAIcAC4ZtswA4vDw/ELio3BsYke3bgHsl7VxaFx0GfHWl00dExKSZPt4GtpdJOgq4AJgGnGb7WknHAYO2FwCnAmdJWgzcRVM0AJB0I7AxsI6k/YA9bf8MeAPwWWB94BvlERERPaIx/pBvnYGBAQ8ODvY6RsRKm4xeNpPxq9qGHG3I0KYcNUhaaHtgvO1af1M5IiLqSEGIiAggBSEiIooUhIiIAFIQIiKiSEGIiAggBSEiIooUhIiIAFIQIiKiSEGIiAggBSEiIooUhIiIAFIQIiKiSEGIiAggBSEiIooUhIiIAFIQIiKiSEGIiAggBSEiIorpvQ4Q0W39NHduxOqY0BmCpL0kXSdpsaRjRli/rqRzyvrLJM3tWHdsWX6dpBd1LL9R0iJJV0sanIwPExERq27cMwRJ04CTgD2AW4ArJC2w/bOOzY4A7rY9T9IhwAnAwZK2Aw4BtgeeAFwo6Um2/1R+7oW275jEzxMREatoImcIOwGLbS+x/RBwNjB/2DbzgTPK83OB3SWpLD/b9oO2bwAWl/eLiIiWmUhBmA3c3PH6lrJsxG1sLwPuAWaO87MGviVpoaQjVz56RERMpl7eVN7V9lJJmwHflvQL298bvlEpFkcCbLnllrUzRkT0jYmcISwFtuh4PacsG3EbSdOBGcCdY/2s7aF/fwucxyiXkmyfYnvA9sCsWbMmEDciIlbFRArCFcC2kraWtA7NTeIFw7ZZABxenh8IXGTbZfkhpRXS1sC2wOWSNpC0EYCkDYA9gWtW/+NERMSqGveSke1lko4CLgCmAafZvlbSccCg7QXAqcBZkhYDd9EUDcp2XwR+BiwD3mj7T5IeB5zX3HdmOvB529/swueLiIgJktegHjcDAwMeHEyXhVg5beiY1oYMbcnRhgxtylGDpIW2B8bbLkNXREQEkIIQERFFCkJERAApCBERUaQgREQEkIIQERFFCkJERAApCBERUaQgREQEkIIQERFFCkJERAApCBERUaQgREQEkIIQERFFCkJERAApCBERUaQgREQEkIIQERFFCkJERAApCBERUaQgREQEMMGCIGkvSddJWizpmBHWryvpnLL+MklzO9YdW5ZfJ+lFE33PiIioa9yCIGkacBKwN7AdcKik7YZtdgRwt+15wInACeVntwMOAbYH9gI+JWnaBN8zIiIqmsgZwk7AYttLbD8EnA3MH7bNfOCM8vxcYHdJKsvPtv2g7RuAxeX9JvKeERFR0fQJbDMbuLnj9S3Ac0bbxvYySfcAM8vyS4f97OzyfLz3BEDSkcCRAFtuueWYQfVejbl+Ivxur/Z7rG6ONmRoS47JyODVf4spkQHakaMNGaAdOdryezpkIgWhp2yfApwCMDAw0IL/hLEyJvN/1ojorolcMloKbNHxek5ZNuI2kqYDM4A7x/jZibxnRERUNJGCcAWwraStJa1Dc5N4wbBtFgCHl+cHAhfZdll+SGmFtDWwLXD5BN8zIiIqGveSUbkncBRwATANOM32tZKOAwZtLwBOBc6StBi4i+YLnrLdF4GfAcuAN9r+E8BI7zn5Hy8iIiZqQvcQbH8d+PqwZe/qeP5H4KBRfvZ44PiJvGdERPROeipHRASQghAREUUKQkREACkIERFRtL5j2spIJ6iIiFWXM4SIiACm2BlCRMSapG1XNXKGEBERQM4QuqJtVT8iYiJSEKaoFKWIWFm5ZBQREUAKQkREFCkIEREBpCBERESRghAREUAKQkREFCkIEREBpCBERESRghAREQDIXnN6tEq6HbhpNd5iU+COSYqzOtqQow0ZoB052pAB2pGjDRmgHTnakAEmJ8dWtmeNt9EaVRBWl6RB2wPJ0Y4MbcnRhgxtydGGDG3J0YYMtXPkklFERAApCBERUfRbQTil1wGKNuRoQwZoR442ZIB25GhDBmhHjjZkgIo5+uoeQkREjK7fzhAiImIUKQgREQGkIET0jKSZvc4AIGkfST39LmjRsdih1xl6KQUhWkHSE3qdoQculfQlSX8jST3McTDwf5I+KOkpPcrQlmPxKUmXS3qDpBk9zPEoSZtV21e/3lSWdIrtIyvs52m2f1qerw38M7ATcA3wb7Yf6HaGsu+jgLNt3yFpHnAa8DTgOuDvbC+qkWOMfL+yvWWF/XwNGPV/etv7djtDRxYB/w94DfBs4IvAZ23/slaGjiwbA4cCr6Y5PqcDX7B9X6X9t+lYbFtyHARcDpxu+9uV9v0XwxcBC4Edab6v7+rq/qdyQRjh4D66CviJ7TkVMlxp+5nl+UeAmTS/bPsBM20f1u0MZd/X2t6+PD8f+C/b50l6AXC87efVyDFGvpttb1FhP88vT/cHNgf+u7w+FPiN7Td3O8MouV5YsmwA/AQ4xvaPK2eYCbwS+Efg58A84OO2P1E5RxuOxTSa39GPA/fSfGe8w/b/dHm/j/Dnw/PMAW4BbPuJ3dw/tqfsA/gTsAS4oeMx9PqhShmu6nh+NbB2eS7gpxWPxXUdz68Ytq5ajjHy/ary/gYnsqzLGWYCRwODwPk0RWo6MADcUDHHvsB5wCLg7cBmZfljgBv77Fg8DTgR+CVwEvDMsvwJwE0V9v9W4JvADh3Lqn3+6atRS9YES4Ddbf9q+ApJN1fKMEPS39Lcr1nX9sPQlHpJNU/PzpX0WeA44DxJ/0jzJbAb8GfHpxskfYKRL9cI2KRGhg4bSHqi7SUAkram+Yu0ph8DZwH72b6lY/mgpE9XzHEAcKLt73UutP2ApCMqZWjLsfgE8F80ZwN/GFpo+1ZJ/9rtndv+iKRzgBPLd9S7GeMS52Sb6peM3gj8wPZPRlj3D65wKizp9GGLjrH9G0mbA5+zvXu3M3RkeRXwemAbYF3gZuArwAm276mw/8PHWm/7jG5n6MiyF00P0CU0BWkr4HW2L6iYQZ7Kv4ArIcfiz0naF3gHMNf25lX2OZX/G0ja2fal/Z6hTTlGImk9YB/bX6q833WBoVY1v7D9YOX9zwL+CdgeWG9oue3dKufYmeYv478E1gGmAffb3rhihrYci22B9wPbDcvR3Wv3y/e/vzvuU0haH9jG9jU19j/Vm51+qtcBaEcGaE8OoLlpV5oYnkVzE+3gyvt/DM318qPKGeSWkl5SMwPwOeAXwNbAe4EbgSsqZwD4JM1N9f8D1gf+jub6eU1tORanAycDy4AXAmeyvOFBDStclrL9h1rFAKZ+QYiWkfR8Sf9J8wt/BLAHsLXtAytHOR14CNilvF4K/FvlDDNtnwo8bPsS26+huadTne3FwDTbf7J9OrBX5QhtORbr2/4OzdWTm2y/B3hxD3L0xFS/qfxESQtGW+k6bc7bkKEVOSTdQnMD+2Tgbbbvk3SDK/XFGGYb2wdLOhQevYFau0PUw+Xf2yS9GLgVGK2pdDc9IGkd4GpJHwRuo/4fi205Fg+WXtv/V/ruLAU2rLj/p0j66QjLRdMW5Wnd3PlULwi3Ax9JBqAdOc6ladt9MPAnSV+lYguKYR4q12ebNsDSNkDVewjAv5XesG+luYa/MdCLfhCvpLlvcFTZ/xY0LY9qasuxOJqmue2bgPfRnKWM2Rhikt0A7FNxfyuY6jeVr7K9Y79naFkOAS+guWb9N8AMmktHX7f9+4o59qC5Xrsd8C3gecCrbH+3VoaI4Xr9ezrVzxBu6HUA2pEBWpKjNC28GLi4DOWxF3AIzU3vTSvm+LakK4GdaU7Hj7ZdZUL1MfpjDGV7U6Uci8bJ0dXLEyVDW45FW4Y0+WGl/Yxoqp8h7D/Wene5G3pbMrQpx2gkrd/ZEaiL+3mK7V9IeuZI621fWSFDK/pjSNpqnBzDh1DoRoa2HIvnj7Xe9iWVcrxlnBwf7er+p3hBeIRmuIirhxZ1rHZpyTDlM7QlR2nj/S/AXcBHgc8AfwVcDxxhe7BChlNsHynp4hFWu3a795LpMT26sd6ZYStgW9sXlnsr011pYLthOdpwLNYHtrR9XQ/2PfR7+g2ae1orNHSw/d6uBujGeBhtedDcwDybZnyUdwLz+jFDW3IAPwCOBN5G03rjIJrOP3sAl1XMsRbwvF78dxiWYxfgZ5RxnICnA5/qQY7X0rT5v7683hb4Tp8ei31oRgC+obx+BrCg4v6fAXyApiicSjMCrKrtv/YB78WDZoyalwFfLV9Kz+/HDL3OAVzd8XzxaOsqZbmq5v5GyXAZTYuezgEQr+lBjqtpeih35ljUp8diIU1Dh54di479PpemxdXPgX1r7LNfOqb9EbiHZhjbDenokt5nGXqd45GO5/eOsa6G70g6oAd9D1Zge/ggi3/qQYwHbT809ELSdHrQHLglx+Jh//m4XtWPRRnKY0dgB5qhr39bY79TupWRpN1oWrDsBFwIfMwVrlO3LUOLcgx1uhGwTUcHHAFVxorp8DrgLcAySX9kecefauP3ADdLei7g0uLqaJq/Bmu7RNI7gPVLc9w3AF+rnKEtx+JaSS8DppV7Xm8CflRr55JeA7yU5g+1c4GX2q5SDKA/bir/lObSiBlW6V2hSVsbMrQlRxtatZQcArbwCMOi1yRpU+BjlOvENP0hjrZ9Z+Uca9H0Bdmz5LiAZgKlesMut+dYPIam4UPnsXif7T9W2v8jNLMpDv0uDP897Wrz16leEHrepK0NGdqUYyIk/dj2LuNvuVr7WGS7rydUj/bpdfPXKX3JaLQvuaHhlvslQ5tyTFCN+xpXSnq27eojaraoM1Y6pi3P0YqOaaN94UvaguaSb1cLQr/cVO75cMttydCmHGOocdr6HODHkq6X9FNJi0YZVKwbBmlas6wHPJNm2On/o2lyuE6lDAAvoflj4Jvl8fLy+Abw9UoZ2nIsPkwz1tcNwB9o+sh8Bvg9TT+Z6iTNkvQGSd8Hvgs8ruv7nMqXjODRU7CX0YybcznNmDVPdMXOL23I0KYc45F0pe0RexJP4j5GvJ9R6z5GyXApsKvtZeX12sD3be9cK0PZ75+Nn1Pjv8Gw/bXlWAzaHhhvWRf3vxHNfNIvA54E/A9wsO05NfY/pc8Q1Ay3/H6aG6nb2T4A+EPlYtDzDG3KMUFdbwrqZqz7m2j+GjQj3Giv4LE0o3oO2bAsq02Sntfx4rnU/25oy7HYQNKjLd5Uf67t3wKvoZmb44m230ozb0cVU/oeAu0YbrkNGVqRQ9K3bO85gU1fWSHLvjSXCJ5A80u4FU0zx+27ve8OHwCuKsNoCPhr4D0V9z/kCOC0Mvw0wO9ovpRqasuxeDPwXUmdc20fWXH/x7J8sMcvSDqn4r774pJRz4dbbkOGNuQY6dJEr0j6Cc1Y9xfa3lHSC4FX2D6ico7Nae5nQDN8x69r7n9YlhkAI3TMqrX/VhwL9Xiu7ZLhiTSF4VCaoUTeDZxn+5dd3e9ULwidynXJF9Ec5BfZrjbccpsy9CpH+avrbaOtd8URV4euC5fCsKPtRyT9xPbTa2WImAhJT6X5PT3Y9ryu7qufCkIndQy3LOnL5Zp632WomUPSnTRjKI10j8CuNPJryXIhzSW099PMw/Bb4Nm2n1srQ8Sq6lZfnal+D2FUXnHs/drDJrQmQ+UcN9X80h/HfJpxnd5M09RyBnBcTxNFTFxX+ur0bUEYpg2nSW3IAN3N0dOB5DrZvh9A0sbUH7fnUZKm0bQvf/R3sRdDapSWRXOH5Tizcoa2HIvZNDeTO3N8r3aOcXTl9zQFIWp6Ra8DDJH0OuC9NGcJj1AGt6PimZqkf6C5Wfgblo/2aqDrPYSH5TgL2IZmGOyhEUYNVCsILToWJ9C0xPsZKx6LthWErkhBaLThL9c2ZIDu5rhU0kh/2fRipNG3AU91pXmUR3E08OTaA7iNYICmb0ovz1Lbciz2KzmqtyxaSV35PZ3SHdPGMqx97z/3a4aaOWxvZHvjER4bVS4G0AxH0OtOeTc5e++UAAAQy0lEQVTTzE3Ra9cAm/c4Q1uOxRJg7V7tXNLJ5TLmeLrSV6efWxn9yvaW/Z6hZg5JfzHWett3dTtDR5YdgdNpZup69K/BSsOAD02kvj3wZOD8YRm6OpF6R46hAd02ohk76PJhObo+oFuLjsXQIHuzaabv/A6V/78oOd5O0xHu3bY/X2OfnXLJKGq6g2b2p2Xldedpb9Xr98B/AhcBi6g/W9tG5d9flcc6LB/IreZfaB+uuK/RtOVYDE0WtRBYMGxdtRy2PyTp88BHJR0BnEzH/5/d7qszpQuCpNEG5xKVTgvbkKFFOT4OvBD4IfAF4Ac9vG69tu23jL/Z5LP9XgBJB9n+Uuc6SQdVzHFJ2ecJtle4VFhurnZ1qOWSoS3H4oyyz6Ntf2xYjqNr5ShZlko6HzieZjTazpvsXS0IU/qSURkXZVS2X9gPGVqWo3P4jJ1oZsY62fYNNfbfkePfgRtpmpx2Xhqoednqz0YUrT3K6Bg5flpjPoRxMrTlWFQbckXS9jRnBbcCb7Z9W439DpnSZwhjfclJes5o66ZahpblMHCxpKtoxmp5H83495+plaE4tPx7bGc8Kly2krQ3zVhSsyV9vGPVxiy/nNZ1kl5PM3/yE7XiXBAbUWke4RYdi0NphpzeWlLnJaONgGp/JNAMQnm07W9V3OejpnRBGMeXgF7f0G1DBqiUQ9IGND2EDwZm0Zz+PqsXnY9sb117nx1upblmvS/NNesh99H0nK7l8zST4bwfOKYzR8UzpbYcix8Bt9EMY/KRYTlqTZwE8IyRmrxK2hU41PYbu7nzKX3JaCySbra9Rb9nqJlD0v00ZwNnl3+HTyBebXC7kUjavOYIm5LWtv1wrf2Npde9hNt0LNqitIR7GXAQzUxu/2P7E93cZz+fIbShErYhA9TL8aWyryeXx/AMPS0IwKnAiyvubydJ72H5MAlDHfSqjmsl6SiauQd62Uu4Lcdif+AEYLOSoWqnSUlPormceShNq7xzaP5wr3OPbyqfIWj0ibMF7Ga76zMhtSFDm3LEcpJ+QXNZZCHLh0mgdm9dSYuB5/Syl3DLjsU+tn9ec78d+38E+D5whO3FZdmSWoVxqp8hjNXOulYb7DZkGG9fVXJ0dEIaUY1OSJJ2s31Reb51Z+smSftXvmx1j+1vVNzfaNrQS7gtx+I3vSoGxf40jS0ulvRNmsur1Ya1mdJnCNEukt49xmrb7vrw053NCoc3MazdzFHSB4BpNJfKOpu+XlkrQ8lxKj3sJVwytOVYfIxmGI+vDMtR9XJmRwOMQ2lm9juTZsa0rrY+mtJnCJIWMcb18RrtrNuQoS05hjohjUTSP3Z7/0O7GuX5SK+7bai570DHMtN8AdQ0Ui/h2tpyLDamGeOqc+7v6ve3yvDsnwc+L+mxNDeW/5mm307XTOkzBElbjbXe9k39kKFNOUZTcTyl1pwhtI2kDQFccZ7vWJGk9YC/B+bRDKtyqu16/TGmckEYiaRNgTt7OdRvGzK0KUfJUqvp6+9oxrYX8FcsH+dewK62H9vtDB1ZZtDMAfDXZdElwHGuPMm9mjl7zwKGBh+8AzjM9rUVM7TlWMwBPgE8ryz6Pk1HsVsq7f8c4OGy371pZhmsNnTGlC4IknYGPkDT0/B9NP/Tb0oz7Pdhtr/ZDxnalGM0Fc8Qnj/W+qHxfWqQ9GWaoafPKIteCTzd9v61MpQcPwL+xfbF5fULgH93xfmlW3Qsvk1zqeassugVwMtt71Fp/4ts71CeTwcur3rWanvKPmh6QO5Jc/3tbmDnsvwpwFX9kqEtOWh6fd47wuM+YFmv/3+p/QCunsiyCjl+MpFlfXIsepoDuHKs191+TPUJcqbb/pabURR/bftSANu/6LMMrcjhsSfIqdLAQdJ8SW/seH2ZpCXlcWCNDB3+UIYkGMryPOAPlTMALJH0Tklzy+NfaSaKqaktx+JOSa+QNK08XgHU7AvxdEn3lsd9wNOGnku6t9s7n9KtjFhxnPvh/3PVulbWhgxtytFr/0TTznvIusCzgQ1oJsw5t2KW1wNnlOvnormc96qK+x/yGpr5pYda0ny/LKupTcfiE8CJNL8XPwJeXWvntqfV2tdIpvo9hD8B99P8D7Y+y6dMFLCe7a7PA9CGDG3K0WuSrrD97I7Xn7R9VHl+qe2de5BpYwDbXf8LsO1yLHprSheEiOEkLbY9b5R119vepmKWTYDDgLmsOKhclekaO3IMAO8YIUfN+RDaciy2Bv5hhBxdn060Dab6JaOI4S6T9FrbK8y/IOl1NHMK1/R14FJ6M41np88Bb+9xjrYci6/QDHL4tR7n6ImcIURfkbQZy4clGBoW4Vk09xL2s/2billa0RFO0g9s7zr+ll3N0JZjcZntahNGtU0KQvQlSbsB25eX17oMeFc5w5uB3wP/S4+m8Sw5dqcZM+c79Gj8nhYdi5cB29IMEdGzMZV6JQUh+l4ZSOxvaWakqjYfQmn+ejzwO5a39LLrzwHw3zT9Ua6lYz4E29VaGrXoWLyfplPc9ax4LGqPqdQTKQjRlyStQzMZzsuAFwFfppmR6msVMywBdrJ9R619jpLjOtvDJyyqnaEtx2IxsJ3th3qZo1emese0iBVI2lPS6TRTEh5AM6zwXbZfXbMYFItZ3vy3l34kabseZ2jLsbgG2KTXIXolrYyi33yTpuPVri6T45Qx8HvhfuBqSRez4vXqqk0tgZ1LjhtKjqFpI2tOodmWY7EJ8AtJVwzLkWanEVPQM2l6Kl9YLlOcTTMxSy98pTx6ba9eB6A9x2KsSZymvNxDiL4l6bk0rWsOAH5CMyPVKb1NFdE7KQjR9yStBfw/4JCaLWsi2iYFIfqKpDE7P/VLe/OIkaQgRF8pNy1H0zftzSNGkoIQUZmkzWluXj4CvItmMLUDgJ/TTNd4W6UcWwAfAmYD3wA+ZPvhsu4rtverkKEtx2Jj4FhgDvAN25/vWPcp22+okaPX0g8hApC0R5k+sYbPAj8DbgYuppmf4m9omsN+ulIGgNOA79J8CT8euETSzLJuq0oZPks7jsXpNM1tvwwcIunLktYt66oPid4rOUOIvlLGMPo08ASaZo4nsPzL4Pga4/dIusr2juX5CnNJS7ra9jO6nWGkfZXZwY4F9gW+VGOwuRYfi3+hKUz7At9uw8B7NaQfQvSbjwBHAj8G9i7/HmP7kxUzdJ6ZnznGum5bW9J6tv8IYPu/Jf0auIBmBrka2nIs1pW0lu1HAGwfL2kp8D1gw4o5eiqXjKLf2PZ3bT9o+yvA0srFAOCrkjYsYf51aKGkecAvK+b4L2CFoZ5tXwgcRDOEQw1tORZfA1ZoUGD7s8Bbgb4Z1yiXjKKvlN7Jb+tY9CGayWGAukM+R7RNCkL0lTKw3WiqDvk8EknPbENfCEkvsf2/Pc7QlmPRihw15B5C9BXbr+51hnG8Hnhtr0MAz6aZrKaX2nIs2pKj63KGEH1H0jTgsUNj75e5EV4FvNn2X/YyW0Qv5aZy9BVJhwB3AT+VdImkPYElNC2OXl4xxwxJB0t6S3kcLKn6OPySNpa0zQjLqw19LWnz0kENSbMk7S9p+/F+rkKuf+91htpyhhB9RdI1wH62F5dxjX4MHFh5prTDaHrnfgtYWhbPAfYA3mt7ePPLbuV4KfAfwG+BtYFX2b6irKsy6b2k1wHH0PQDOYHmTO0aYFfgg7ZP7XaGkuPjwxfRTKV5JvRkXoaeSEGIvjL8i07SNbafWjnDdcBzbP9u2PLHApfZflKlHFcDe9u+TdJONF9+x9o+r7PDWJczLKJp+ro+cBMwz/avy7G4uGLHtJuBS2iKtMriD1NapNk+o0aOXstN5eg3m0l6S8frTTpf2/5ohQxi+UTynR5h+ZdRDdOGxgqyfbmkFwL/W8Y4qvWX4sO2HwAekHS97V+XPHdLqvnX6nbA+2gmC3qb7VslvbtfCsGQFIToN58BNhrlda0voOOBKyV9i2YMH4AtaS4Zva9SBoD7JG1j+3qAcqbwApohPWpdw7ektcugei8eWihpPSre47R9H/CPkp4FfE7S+TX33xa5ZBRRSHr20DX0Cvt6LPAimpFGobmXcIHtu2vsv2R4OnC/7cXDlq8NvNT25ypk2BK41fayYctnA39Zek5XJUnAG4BdbL+i9v57KQUh+pqk7Wim0TwU+J3tgR5HepSkH9veJTnakaFNOboll4yi70iay/Ii8DDNUM8Dtm/sXaoRrdfrAEUbcrQhA7QnR1f03TWy6G+SfgycT/PH0AG2nwXc18JiAPXuaYynDTnakAHak6MrUhCi3/yG5iby44BZZdmU/iWPmKgUhOgrZVrIHYCFwHsk3QA8trTDb5uaTVDH0oYcbcgA7cnRFSkI0Xds32P7dNt70kyP+C7gxNI5qesknVzm8B3PK6d6jjZkaFOOXktBiL5m+ze2P2H7eTTDJdSwBFgo6WVjbWS725PUtCFHGzK0KUdPpdlp9BVJC8Zab3vfSjlmAx8FNgVOpumlPJSh2iQ9bcjRhgxtytFLaXYa/WYXmt7BXwAuo0fXhG0vLb1hjwf2YfmXj4FqXz5tyNGGDG3K0UspCNFvNqcZIuJQ4GU0TVC/YPvaWgHK0M4nA7cCOw2NJ1RbG3K0IUObcvRaLhlF35K0Lk1h+BDNsNOfrLTfnwNH2/5Wjf21OUcbMrQpR6+lIETfKYXgxTTFYC6wADjN9tKxfm4y92/7wRGW7wocavuN/ZKjDRnalKPXcsko+oqkM4GnAl+nOSuo3mqk84tH0o40l64OAm6g7jXznudoQ4Y25ei1FIToN68A7geOBt7UDGwJlDkKbE+kLfpqkfQklo+ldAdwDs3Z+gu7ve+25WhDhjbl6LVcMoqoTNIjwPeBI4aGnpa0xPYT+y1HGzK0KUevpWNaRH37A7cBF0v6jKTd6U3z1zbkaEOGNuXoqZwhRPSIpA2A+TSXKXajmdP4vNotXdqQow0Z2pSjV1IQIlqgzKB2EHCw7d37OUcbMrQpR00pCBGVlfmC/x6YBywCTh0+hWS/5GhDhjbl6LUUhIjKJJ1DM1Pb94G9gZtsH92POdqQoU05ei0FIaIySYts71CeTwcut/3MfszRhgxtytFraWUUUd/DQ096fFmiDTnakKFNOXoqZwgRlUn6E03nOGiaNq4PPEDFznFtydGGDG3K0WspCBERAeSSUUREFCkIEREBpCBERESRghDRQZIlfaTj9dskvaeHkSKqSUGIWNGDwP6SNu11kIjaUhAiVrQMOAV48/AVkvaRdJmkqyRdKOlxZfl7JJ0h6fuSbpK0v6QPSlok6ZuS1i7bPUvSJZIWSrpA0uPrfrSIsaUgRPy5k4CXS5oxbPkPgJ1t7wicDfxTx7ptaEbH3Bf4b+Di0vP1D8CLS1H4BHCg7WcBpwHHd/djRKyczJgWMYzte8tUm2+i+UIfMgc4p/xlvw7N9IpDvmH7YUmLgGnAN8vyRTTzNj+ZZurOb5dZ2qbRjL8f0Ro5Q4gY2X8ARwAbdCz7BPDJ8pf/64D1OtY9CGD7EeBhL+/x+QjNH14CrrX9jPLYwfae3f4QESsjBSFiBLbvAr5IUxSGzACWlueHr+RbXgfMkrQLgKS1JW2/2kEjJlEKQsToPgJ0tjZ6D/AlSQtpJmKfMNsPAQcCJ0j6CXA18NxJyhkxKTKWUUREADlDiIiIIgUhIiKAFISIiChSECIiAkhBiIiIIgUhIiKAFISIiChSECIiAoD/D0RWJqkQOaYSAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe5cb77a860>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plot_local_contrib(row, local_glm1, X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Use LIME to generate descriptions for a local region with a practical sample\n",
    "Using a previously-existing local sample based on clusters, deciles, or other more natural segments to create LIME explanations is computationally cheaper and perhaps more straightforward than using a simulated perturbed sample, but it does have one major drawback. If the sample is too large, the explanatory linear model maybe not be accurate enough to explain all predictions in the sample. The remaining sections of this notebook will explore the idea of generating LIME explanations using a practical sample. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create a local region based on values of SEX and merge with GBM model predictions\n",
    "Instead of using a perturbed simulated sample, a linear model will be fit on all women in the test set, and the sample is not weighted by distance from any one point. A few lines of the all female sample are displayed directly below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gbm prediction progress: |████████████████████████████████████████████████| 100%\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<table>\n",
       "<thead>\n",
       "<tr><th style=\"text-align: right;\">  ID</th><th style=\"text-align: right;\">  LIMIT_BAL</th><th>SEX   </th><th>EDUCATION  </th><th>MARRIAGE  </th><th style=\"text-align: right;\">  AGE</th><th>PAY_0                  </th><th>PAY_2                  </th><th>PAY_3                  </th><th>PAY_4                  </th><th>PAY_5                  </th><th>PAY_6                  </th><th style=\"text-align: right;\">  BILL_AMT1</th><th style=\"text-align: right;\">  BILL_AMT2</th><th style=\"text-align: right;\">  BILL_AMT3</th><th style=\"text-align: right;\">  BILL_AMT4</th><th style=\"text-align: right;\">  BILL_AMT5</th><th style=\"text-align: right;\">  BILL_AMT6</th><th style=\"text-align: right;\">  PAY_AMT1</th><th style=\"text-align: right;\">  PAY_AMT2</th><th style=\"text-align: right;\">  PAY_AMT3</th><th style=\"text-align: right;\">  PAY_AMT4</th><th style=\"text-align: right;\">  PAY_AMT5</th><th style=\"text-align: right;\">  PAY_AMT6</th><th style=\"text-align: right;\">  DEFAULT_NEXT_MONTH</th><th style=\"text-align: right;\">  p_DEFAULT_NEXT_MONTH</th></tr>\n",
       "</thead>\n",
       "<tbody>\n",
       "<tr><td style=\"text-align: right;\">   4</td><td style=\"text-align: right;\">      50000</td><td>female</td><td>university </td><td>married   </td><td style=\"text-align: right;\">   37</td><td>use of revolving credit</td><td>use of revolving credit</td><td>use of revolving credit</td><td>use of revolving credit</td><td>use of revolving credit</td><td>use of revolving credit</td><td style=\"text-align: right;\">      46990</td><td style=\"text-align: right;\">      48233</td><td style=\"text-align: right;\">      49291</td><td style=\"text-align: right;\">      28314</td><td style=\"text-align: right;\">      28959</td><td style=\"text-align: right;\">      29547</td><td style=\"text-align: right;\">      2000</td><td style=\"text-align: right;\">      2019</td><td style=\"text-align: right;\">      1200</td><td style=\"text-align: right;\">      1100</td><td style=\"text-align: right;\">      1069</td><td style=\"text-align: right;\">      1000</td><td style=\"text-align: right;\">                   0</td><td style=\"text-align: right;\">              0.144991</td></tr>\n",
       "<tr><td style=\"text-align: right;\">   8</td><td style=\"text-align: right;\">     100000</td><td>female</td><td>university </td><td>single    </td><td style=\"text-align: right;\">   23</td><td>use of revolving credit</td><td>pay duly               </td><td>pay duly               </td><td>use of revolving credit</td><td>use of revolving credit</td><td>pay duly               </td><td style=\"text-align: right;\">      11876</td><td style=\"text-align: right;\">        380</td><td style=\"text-align: right;\">        601</td><td style=\"text-align: right;\">        221</td><td style=\"text-align: right;\">       -159</td><td style=\"text-align: right;\">        567</td><td style=\"text-align: right;\">       380</td><td style=\"text-align: right;\">       601</td><td style=\"text-align: right;\">         0</td><td style=\"text-align: right;\">       581</td><td style=\"text-align: right;\">      1687</td><td style=\"text-align: right;\">      1542</td><td style=\"text-align: right;\">                   0</td><td style=\"text-align: right;\">              0.128193</td></tr>\n",
       "<tr><td style=\"text-align: right;\">  16</td><td style=\"text-align: right;\">      50000</td><td>female</td><td>high school</td><td>divorced  </td><td style=\"text-align: right;\">   23</td><td>1 month delay          </td><td>2 month delay          </td><td>use of revolving credit</td><td>use of revolving credit</td><td>use of revolving credit</td><td>use of revolving credit</td><td style=\"text-align: right;\">      50614</td><td style=\"text-align: right;\">      29173</td><td style=\"text-align: right;\">      28116</td><td style=\"text-align: right;\">      28771</td><td style=\"text-align: right;\">      29531</td><td style=\"text-align: right;\">      30211</td><td style=\"text-align: right;\">         0</td><td style=\"text-align: right;\">      1500</td><td style=\"text-align: right;\">      1100</td><td style=\"text-align: right;\">      1200</td><td style=\"text-align: right;\">      1300</td><td style=\"text-align: right;\">      1100</td><td style=\"text-align: right;\">                   0</td><td style=\"text-align: right;\">              0.325205</td></tr>\n",
       "</tbody>\n",
       "</table>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/plain": []
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds2 = model.predict(test).drop(['predict', 'p0'])\n",
    "preds2.columns = [yhat]\n",
    "practical_sample = test.cbind(preds2)\n",
    "practical_sample = practical_sample[practical_sample['SEX'] == 'female']\n",
    "practical_sample.head(rows=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Train penalized linear model in local region \n",
    "A penalized linear model is trained in the local region defined by women in the test set. Because fit is a concern in this much larger explanatory sample, users should always check the R<sup>2</sup> or other goodness-of-fit measures to ensure surrogate model is accurate in the sample. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "glm Model Build progress: |███████████████████████████████████████████████| 100%\n",
      "\n",
      "Local Positive GLM Coefficients:\n",
      "BILL_AMT5:                1.0399411952809277e-07\n",
      "BILL_AMT2:                1.1999977329358182e-07\n",
      "AGE:                      0.0005425600656944679\n",
      "EDUCATION.high school:    0.0007216688611868718\n",
      "PAY_0.8 month delay:      0.004556460822481314\n",
      "PAY_2.7 month delay:      0.004556460822481664\n",
      "PAY_3.6 month delay:      0.004556460822481671\n",
      "EDUCATION.university:     0.004929360519089897\n",
      "PAY_3.3 month delay:      0.009929773165479217\n",
      "MARRIAGE.married:         0.010735128951809985\n",
      "PAY_6.2 month delay:      0.018087589078182496\n",
      "PAY_5.3 month delay:      0.022758693814452228\n",
      "PAY_2.2 month delay:      0.026627792771502998\n",
      "PAY_4.2 month delay:      0.028646065263953614\n",
      "PAY_2.3 month delay:      0.028779499175080204\n",
      "PAY_5.2 month delay:      0.041320673312054176\n",
      "PAY_5.7 month delay:      0.04168881231553721\n",
      "PAY_4.7 month delay:      0.04168881231553731\n",
      "PAY_6.3 month delay:      0.11704990177038722\n",
      "PAY_0.4 month delay:      0.14562584980767246\n",
      "PAY_0.3 month delay:      0.21475212459213328\n",
      "PAY_0.2 month delay:      0.2253878590232612\n",
      "Intercept:                0.4675657228203593\n",
      "\n",
      "Local GLM R-square:\n",
      "0.93\n"
     ]
    }
   ],
   "source": [
    "# initialize\n",
    "local_glm2 = H2OGeneralizedLinearEstimator(seed=12345)\n",
    "\n",
    "# train \n",
    "local_glm2.train(x=X, y=yhat, training_frame=practical_sample)\n",
    "\n",
    "# coefs\n",
    "print('\\nLocal Positive GLM Coefficients:')\n",
    "for c_name, c_val in sorted(local_glm2.coef().items(), key=operator.itemgetter(1)):\n",
    "    if c_val > 0.0:\n",
    "        print('%s %s' % (str(c_name + ':').ljust(25), c_val))\n",
    "        \n",
    "# r2\n",
    "print('\\nLocal GLM R-square:\\n%.2f' % local_glm2.r2())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The R<sup>2</sup> is quite high for this local sample and linear model. Because the sample is simply the women in the test set and because the model fit is acceptable, the trained linear model and coefficients can be used to understand the average behavior of women in the test set. On average, late payments, particulary `PAY_0`, `PAY_2`, and `PAY_6`, are the most likely to push the GBM model towards higher probability of default values for women."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6. Generate a ranked predictions plot to assess validity of local explanatory model\n",
    "A *ranked predictions plot* can also be used to ensure the local linear surrogate model is a good fit for the model inputs and predictions. A ranked predictions plot is a way to visually check whether the surrogate model is a good fit for the complex model. The y-axis is the numeric prediction of both models for a given point. The x-axis is the rank of a point when the predictions are sorted by their GBM prediction, from lowest on the left to highest on the right. When both sets of predictions are aligned, as they are below, this a good indication that the linear model fits the complex, nonlinear GBM well in the practical sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "glm prediction progress: |████████████████████████████████████████████████| 100%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX8AAAEICAYAAAC3Y/QeAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzs3Xd4VFX6wPHvO2mEJCSkQIAgCb03KTaagiAo9i6KWFZde+8glnVdVtd19aeuBXdtWFZFRUERBAGR3jsECDWFQEIISWbO748pmUxmkkky6e/neXjI3HvuPWcG8s65p4oxBqWUUo2LpbYLoJRSquZp8FdKqUZIg79SSjVCGvyVUqoR0uCvlFKNkAZ/pZRqhDT4q1ojIlNE5MMA3StVREYG4l5VyVtEHheRdyp5nw0iMjyghaskERkuImm1XQ5VfTT4qzI5AtsJEckVkYMiMl1EImu7XBXhKHOB4z1kichPItK1OvIyxrxgjLnZzzI953FtD2PM/Oool48yTBQRq+NzOSYiq0Xk/Ercp9R7UXWfBn/ljwuMMZFAX6Af8Fgtl6cyXnK8hyTgMDDdWyIRCa7JQtUBSxyfSwzwLvCZiDSv5TKpGqDBX/nNGHMQmI39SwAAERknIqscNce9IjLF7VyyiBgRuUFE9ohIhog84e3eIhIiIp+IyJciEioiFhF5VER2iEimiHwmIrFu6SeIyG7HOa/39PEe8oCPgZ6O+0wRkS9E5EMROQZMrErenk1ZInKWiCwWkWzH5zNRRG4FrgUedtS6v3WkdW8+ChORf4jIfseff4hImOPccBFJE5EHROSwiBwQkRvd8hwrIhtFJEdE9onIg358LjbgPSAc6OB5XkS6ich8x/vYICLjHce9vhdV92nwV34TkSTgPGC72+HjwPXYa47jgNtF5CKPS88CugDnAE+LSDeP+4YDXwMngSuMMQXAXcBFwDCgNXAEeN2Rvjvwf8AEx7k47DV6f95DJPZgtcrt8IXAF4738FGg8haRdsAPwGtAAvYvzdXGmLcd+bxkjIk0xlzg5fIngNMc1/QBBgFPup1PBKKBNsBNwOtuNfZ3gT8ZY6Kwf8n94sfnEgzcDOQC2zzOhQDfAnOAFo7P5yMR6eLne1F1kAZ/5Y+vRSQH2Iu9yWSy84QxZr4xZp0xxmaMWQt8gj1ounvGGHPCGLMGWIM9mDk1A34EdgA3GmOsjuO3AU8YY9KMMSeBKcBljiB1GfCdMWaB49xTgK2c9/CgiGRj/+KKBCa6nVtijPna8R5OBDDva4CfjTGfGGMKjTGZxpjV5ZTT6VpgqjHmsDEmHXgG+xeOU6HjfKExZhb2oN3F7Vx3EWlmjDlijFlZRj6nOT6Xg8DVwMXGmKOeabB/Zi8aYwqMMb8A3znSq3pKg7/yx0WOWuRwoCsQ7zwhIoNFZJ6IpIvIUeyBM97j+oNuP+dhDyROpwG9sQcW91UG2wFfOZoZsoFNgBVoib3GvdeZ0BhzHMgs5z1MM8bEGGMSjTHjjTE73M7t9UgbqLzbYv9Sq4zWwG6317sdx5wyjTFFbq/dP9dLgbHAbhH5VUROLyOf3x2fS7wx5jRjzM8+yrLX0TTkXp42/r4ZVfdo8Fd+M8b8ir2jdJrb4Y+BmUBbY0w08CYgFbjtHOAvwFwRael2fC9wniMwOf80McbsAw5gD6wAiEhT7M0vleW5tG2g8t6Ll/ZzH3l62o/9S8jpFMexchljlhljLsTeRPM18Jk/15VTlrYi4h4vTgH2ObOs4v1VLdDgryrqH8AoEXE23UQBWcaYfBEZhL2po0KMMS9h/xKZKyLOp4Y3gecd7eaISIKIXOg49wVwvqMzNRSYSmD/Lwcq74+AkSJyhYgEi0iciDg7yw8B7csowyfAk46844GngXLnRDg6y68VkWhjTCFwjPKbxMqzFPuTxcOOjvnhwAXAp47z5b0XVQdp8FcV4mh//g/2YARwBzDV0SfwNJWsZRpjnsVeS/3ZMbLmVexPFHMc9/4dGOxIuwH4M/YvjAPYO2QDOSEpIHkbY/Zgb355AMgCVlPc3/Eu9nb5bBH52svlzwHLgbXAOmCl45g/JgCpjtFLt2HvP6g0Rwf8Bdg7+zOAN4DrjTGbHUnKey+qDhLdzEUppRofrfkrpVQjpMFfKaUaIQ3+SinVCGnwV0qpRqjOLmIVHx9vkpOTa7sYSilVr6xYsSLDGJNQXro6G/yTk5NZvnx5bRdDKaXqFRHZXX4qbfZRSqlGSYO/Uko1Qhr8lVKqEaqzbf7eFBYWkpaWRn5+fm0XRQVYkyZNSEpKIiQkpLaLolSjUK+Cf1paGlFRUSQnJyNSkYUjVV1mjCEzM5O0tDRSUlJquzhKNQr1qtknPz+fuLg4DfwNjIgQFxenT3RK1aB6FfwBDfwNlP67KlWz6l3wV0qphmzx9gx2ZRyv9nw0+FfQ888/T48ePejduzd9+/Zl6dKltV0kl+nTp7N/v1+bPblMmTKFNm3a0LdvX3r27MnMmTOrVIbhw4fr5DylquCad5YyYtr8as+nXnX41rYlS5bw3XffsXLlSsLCwsjIyKCgoMDv64uKiggODvb5uqqmT59Oz549ad26dfmJ3dx33308+OCDbNq0iSFDhnD48GEsluJ6QaDLqZSqfVrzr4ADBw4QHx9PWFgYAPHx8a5Am5ycTEZGBgDLly9n+PDhgL1mPWHCBM4880wmTJjA9OnTGT9+PGeffTbnnHMOxhgeeughevbsSa9evZgxYwYANpuNO+64g65duzJq1CjGjh3LF198AcDUqVMZOHAgPXv25NZbb8UYwxdffMHy5cu59tpr6du3LydOnGDFihUMGzaMU089ldGjR3PgwIEy31+3bt0IDg4mIyODiRMncttttzF48GAefvhhjh8/zqRJkxg0aBD9+vXjm2++AeDEiRNcddVVdOvWjYsvvpgTJ04AYLVamThxout9vfLKK4H9x1BKVUm9rc498+0GNu4/FtB7dm/djMkX9PB5/txzz2Xq1Kl07tyZkSNHcuWVVzJs2LBy77tx40Z+++03wsPDmT59OitXrmTt2rXExsby5Zdfsnr1atasWUNGRgYDBw5k6NChLFq0iNTUVDZu3Mjhw4fp1q0bkyZNAuDOO+/k6aftuyhOmDCB7777jssuu4x//etfTJs2jQEDBlBYWMhdd93FN998Q0JCAjNmzOCJJ57gvffe81nOpUuXYrFYSEiwrwmVlpbG4sWLCQoK4vHHH+fss8/mvffeIzs7m0GDBjFy5EjeeustmjZtyqZNm1i7di39+/cHYPXq1ezbt4/169cDkJ2d7d8/glKqRtTb4F8bIiMjWbFiBQsXLmTevHlceeWVvPjii0ycOLHM68aPH094eLjr9ahRo4iNjQXgt99+4+qrryYoKIiWLVsybNgwli1bxm+//cbll1+OxWIhMTGRESNGuK6fN28eL730Enl5eWRlZdGjRw8uuOCCEnlu2bKF9evXM2rUKMBeE2/VqpXX8r3yyit8+OGHREVFMWPGDNfIm8svv5ygoCAA5syZw8yZM5k2bRpgH3a7Z88eFixYwN133w1A79696d27NwDt27dn586d3HXXXYwbN45zzz3Xr89YKVUz6m3wL6uGXp2CgoIYPnw4w4cPp1evXnzwwQdMnDiR4OBgbDYbQKnx6hEREWW+roj8/HzuuOMOli9fTtu2bZkyZYrX8fHGGHr06MGSJUvKvaezzd+TezmNMXz55Zd06dLFr3I2b96cNWvWMHv2bN58800+++yzMp86lGrsVu45wiVvLK6x/LTNvwK2bNnCtm3bXK9Xr15Nu3btAHub/4oVKwD48ssv/b7nkCFDmDFjBlarlfT0dBYsWMCgQYM488wz+fLLL7HZbBw6dIj58+cDxV8s8fHx5ObmuvoBAKKiosjJyQGgS5cupKenu4J/YWEhGzZsqPR7Hz16NK+99hrGGABWrVoFwNChQ/n4448BWL9+PWvXrgUgIyMDm83GpZdeynPPPcfKlSsrnbdSjYEz8E8Jns60kDerPb96W/OvDbm5udx1111kZ2cTHBxMx44defvttwGYPHkyN910E0899ZSrs9cfF198MUuWLKFPnz6ICC+99BKJiYlceumlzJ07l+7du9O2bVv69+9PdHQ0MTEx3HLLLfTs2ZPExEQGDhzoupezkzY8PJwlS5bwxRdfcPfdd3P06FGKioq499576dGjck9MTz31FPfeey+9e/fGZrORkpLCd999x+23386NN95It27d6NatG6eeeioA+/bt48Ybb3Q9Df3lL38B4M037f+pb7vttkqVQ6mGrptlD4bqn/QozppcXTNgwADjOV5806ZNdOvWrZZKVPNyc3OJjIwkMzOTQYMGsWjRIhITE2u7WNWmsf37KuUu+dHvAfg+9DH2mzhGPTuvUvcRkRXGmAHlpdOafx12/vnnk52dTUFBAU899VSDDvxKKbtITpBLePkJq0iDfx3mbOdXSjUezSSPXFv1B3/t8FVKqToijAKaSy7pJqba89Lgr5RSdUQz7Au6ZdKs2vPS4K+UUnVEjNiDf7aJrPa8NPgrpVQdEUMuANlUfiKovzT4V5CIcN1117leFxUVkZCQwPnnnw/YV9a88847y7zH8OHD6dKlC3369OHMM89ky5YtVSpTZGT11xKUUoGVnVfgmjS59ZB9cma0o+Z/1Gjwr3MiIiJYv369a/XKn376iTZt2lT4Ph999BFr1qzhhhtu4KGHHip13mq1VrmsSqm6aW9WHn2n/sQVby1h2N/muRapjBFnzV+bfeqksWPH8v339gkZn3zyCVdffXWl7zV06FC2b98O2JeIeOSRR+jfvz+ff/45O3bsYMyYMZx66qkMGTKEzZs3A7Br1y5OP/10evXqxZNPPum614EDBxg6dKhrY5aFCxdW4V0qpaqqyGrjn3O3cfxkEfO3HOazZXv5YHEq2w7ba/rLUo+wOzOPl3/aCkAs9i+Bmmjzr7/j/H94FA6uC+w9E3vBeS+Wm+yqq65i6tSpnH/++axdu5ZJkyZVOtB+++239OrVy/U6Li7OtQ7OOeecw5tvvkmnTp1YunQpd9xxB7/88gv33HMPt99+O9dffz2vv/6669qPP/6Y0aNH88QTT2C1WsnLy6tUmZRSgfH16v28/NNWsvMKeW/RLtfx6PCQEun2ZNl/VzvKftJNNLk0rfay1d/gX4t69+5Namoqn3zyCWPHjq3UPa699lrCw8NJTk7mtddecx2/8sorAfvSDosXL+byyy93nTt58iQAixYtci0eN2HCBB555BEABg4cyKRJkygsLOSiiy6ib9++lSqbUqp8L/+0lTE9Eune2vewzIIi+9pWC7ellzh+9ESh1/QdLPvZbqt4M3Jl1N/g70cNvTqNHz+eBx98kPnz55OZmVnh6z/66CMGDCi9/IZzGWWbzUZMTAyrV6/2er1zzX13Q4cOZcGCBXz//fdMnDiR+++/n+uvv77CZVOqsTpZZOX9RancdFYKIUG+W8ULHc05b/66g63PnVfufbcdzvUr/5ZyhKWmZta3Ckibv4iMEZEtIrJdRB71cv4UEZknIqtEZK2IVK66XIdMmjSJyZMnl2iyCaRmzZqRkpLC559/DtjX01+zZg0AZ555Jp9++ilg/xJx2r17Ny1btuSWW27h5ptv1mWUlaqgdxbu4sUfNvPh77v9Su+s2QdCM47Tmkx221oG7J5lqXLwF5Eg4HXgPKA7cLWIdPdI9iTwmTGmH3AV8EZV861tSUlJrh2sPE2fPp2kpCTXn7S0tErl8dFHH/Huu+/Sp08fevTo4do399VXX+X111+nV69e7Nu3z5V+/vz59OnTh379+jFjxgzuueceAG6++WY8V0hVSpWWe7IIgLyCyo22+/D33azfd9T12ssDuk9tJAOLGLaapErlXVGBaPYZBGw3xuwEEJFPgQuBjW5pDLjmK0cD+wOQb63IzS39+Obc2Qvsa+qXt62jrwXbUlNTS7xOSUnhxx9/LJUuJSWlxA5dzz33HAA33HADN9xwQ6n077zzTpnlUUoFxpNf2/esTn1xHECFVuVPlCwA0k10oIvlVSCafdoAe91epzmOuZsCXCciacAs4C5vNxKRW0VkuYgsT09P95ZEKaXqjb/N9n8C5xmWDRQZC5tMu2osUbGaGud/NTDdGJMEjAX+KyKl8jbGvG2MGWCMGZCQkFBDRVNKKbtA75+VebzAr3SxHOPaoLnMsQ0gjyYBLoV3gQj++4C2bq+THMfc3QR8BmCMWQI0AeIrk1ld3XlMVY3+u6q6oLb+F04M/pGmcpJ/FF0KwK1D21d7noEI/suATiKSIiKh2Dt0Z3qk2QOcAyAi3bAH/wq36zRp0oTMzEwNFA2MMYbMzEyaNKmZGo9SdclZlnX8Oegb5lhPZatpW/4FAVLlDl9jTJGI3AnMBoKA94wxG0RkKrDcGDMTeAD4t4jch/3LdaKpRAR3jpzR/oCGp0mTJiQl1cwoB6V88bfZxz165eQXEtWk5IzdvVl5WCzl3y2RTN4JmcZe04IHCm93u3/1V3ADMsnLGDMLe0eu+7Gn3X7eCJxZ1XxCQkJISUmp6m2UUipgxv5zIQsfPrvEsSEvlb/5ehBW/hryb0IoYlLhQ+TUwJIO7nRhN6WUqoK9WScqdd3EoNkMC1rLs0UT2GlaB7hU5dPgr5RSHsprdjFV7Bq+2LKQx4I/5ndbNz6wnlule1WWBn+llPLiRIHVNeM3kNrJQV4IeZc1pgN/KrgPU0thWIO/Ukp5Mej5n+k5eXZA7ynY+EfIGxQSzB0F93C0BjZt8aX+ruqplFIB5r4WT04ZtX7PVqEjxwt489cd5d7/0eBP6GfZzmOFN3GI2MoWMyC05q+UUn6Y+P4fDHnpF6/nJs/cwFsLdpZ5fX/Zyp+Cv+dL61l8Yj27zLQ1MZVJa/5KKeWH+Vt8zy+auab8tSqvC/6ZHBPO5MKJBH4hiYrTmr9SSnkIdM07RQ4w3rKYWdbBfm3RWJGloCtLg79SSlWjAbKZH0MfIZ9Q3reO8euammj20eCvlFIVlJ3nfQ9eT9Hk8lrov0gnhnEFL7DZnFLNJfOftvkrpVQFnfaXuX6kMkwNmU48R7moYCq7TWJ1F6tCtOavlFJlOFFgpfOTP7her9id5dd19wZ/yYVBi3mt6GI2mLq3JpkGf6WUchAvo3DeW7SrxEbtl/7fklJpPHWXVO4N/h8/WAfymvWiMtOueVqXd1BKqTqnIlsxOj0UPIMCE8TkwonlLt8Q3TSkzPPVRYO/UkoFjOGOoG8YEbSGvxddwWGa13aBfNLgr5RSHioz0rIJJ3kp+G0eDpnBd9bBTLeO9vva83u3qkSOVaPBXymlquhMyzrmhT3AFcG/8mrRJdxZeDcnCS33utgIe5rHxnYrcbwmNqrVoZ5KKVUFPWQX74VM47CJ4drCx1hk6+X3tT/fP6waS1Y2rfkrpWrcgaMnSH70e+ZsOFjbRSmhossqBGHl2ZD3OUkIVxc+WaHAD8U1/9qgwV8pVePWpR0F4LPlabVckqowTAn+gP6W7TxZeCNpJqG2C1Qh2uyjlFIVlMARXgp5mxFBa/i4aAQzbWfWdpEqTIO/Ukp5KGthtWhymRX2OM3I4y+FV/OOdWwA8quJLt6SNPgrpZSfosjjh7BHSZCj3FjwEPNs/Wq7SJWmbf5KKeWnK4Lm0VqymFTwYLUGfl3SWSmlalBZg32SJJ27g7/id1s3frH1D2y+NbF7iwcN/kop5Yc/BX1LtOTxROGkgN/bs81fd/JSSqk6IEnSmRD8M99ZT2OHaVPt+dWbZh8RGSMiW0Rku4g86iPNFSKyUUQ2iMjHgchXKaUC6edNhwEwHgssPB38HwpMEK8UXVobxaoWVR7tIyJBwOvAKCANWCYiM40xG93SdAIeA840xhwRkRZVzVcppQJt44FjAPyy+bDrWH/ZyrlBK3i58LJqq/V7tvm3jmlSLfm4C0TNfxCw3Riz0xhTAHwKXOiR5hbgdWPMEQBjzGGUUqqO2p99wvXzuUHLKTBBvOfn5uu+pL44zuc5Z5t/6+gmvHP9ACadWf07fwUi+LcB9rq9TnMcc9cZ6Cwii0TkdxHx+imKyK0islxElqenpwegaEopVXFWW3Gzz2mWjawyncilaYXv8+S4buUnciMijOzeEoul+nt8a2qSVzDQCRgOJAELRKSXMSbbPZEx5m3gbYABAwbU/JQ3pVSDsHH/MbYeyuGifpVrpnEG/zAK6C67ebeSs3hvHtKeLolRJMdF+JW+Jmf6BiL47wPaur1OchxzlwYsNcYUArtEZCv2L4NlAchfKaVKGPvPhQCu4J9XUERokIXgIP8aO5wV/8GWTYSKlRW2zpUuy5BOdXPBt0A0+ywDOolIioiEAlcBMz3SfI291o+IxGNvBtoZgLyVUqpc3Z+ezd2frvI7fZHNvmH7CMtq8k0ICyu4VDPA6e3jKnxNTU72qnLN3xhTJCJ3ArOBIOA9Y8wGEZkKLDfGzHScO1dENgJW4CFjTGZV81ZKKX/NWud774D/Lknlkz+Kuy7zC+3Bf6hlLUts3f3alcuT53DRuiYgbf7GmFnALI9jT7v9bID7HX+UUqpOeeqbDaWOxXGUDpYDfFo4olL37JrYrMLX1GSbv87wVUopL/pZtgOwxtahwtfGRoTy+NiKjfSpaRr8lVK1qO42jZxpWc9JE8wa41/wb59QPKJn5VOjCA2ueHityTZ/Df5KKeVBsHFJ0EJW2jr73d5vs9XdLzJvNPgrpWpRzS9l7I+zLOuJljy+s53m9zWBiP31bZy/UkrVC0fzCgkJFnJPFpFfYKNtbLjXppYLLEs4ZpryhXWo3/e2+hH91005F4BeU+b4X+hqosFfKVXjThRaK3zN3Z+sIq+giHduGFjpfPtMnUNisyYcPJYPQHR4CGsmn1sijWBjsGUTy2xdym3yeWZ8DybPtI8UuuGMdrwwa3OZ6aOahACw8OER5BWU/gy0zV8p1aDd8+nqCl8zc81+15LLZdmVcbzM887AD3D0RCH73BZxA+grO2hnOcws6+AKle/Wof6PCmob25QuiVEVun+gafBXSjUoI6bNr1D6F38oWVsfYlmHzQjzbH0DWKq6R5t9lFIN2uGc/PITuelv2cYW05YsKj5Jq7JaRYczvEsCd47oWGN5as1fKVUvbT2Uw+fL95abbsHWjDLPf7tmv+tnZ3v/BpNcqTL1aF25L4wgizD9xkEMSI6t1PWVoTV/pVS9dO4rCwC4fEBbn2mO5hXy4Odr/L7nxZbfCJcC5lr7lZkuLiKUzOMFpY7PvPMsbDU4XLMqtOavlGqwPliSWqH0Zwet5qBpzg+2QT7TnNY+lnG9W3k9F2QRQvxcNrq21Y9SKqVUNQuhiKGWtSy09sLX5LOzOsbz6a2n12zBqokGf6WUwr5dYzPJY7bN9zyC83ollnhdkzNyA02Dv1KqFtWd4HmeZWmZG7dc0Kc11ww6pYZLVX00+CulGj0LNoYFreU3W0+fs3pDgyw1OgO3umnwV0rVoroRTM+yrKONZPK19Syfadzjft0oddVo8FdKNVj+NslPCvqRo6Ypc2wDqnyv+kKDv1KqUeskaQwPWsN062gKCKnt4tQYDf5KqUbtmqC5FJggpheNLjOde7OPxSKOY/W3AUiDv1KqFtVuW0oUeVwe9CuzbQM5UoG1fO4d2ZkJp7XjyoG+ZxfXdbq8g1Kq0RoftJhIyeftovMrdF10eAjPXtSzmkpVM7Tmr5SqRbXbbHJl0Dx221qwzrSv1XLUBg3+SqlGqZOk0duyiw+sZbf1N1Qa/JVSjdK4oN+xGeFba9XW6kls1iRAJapZ2uavlGp0wijgiqD5LLV1I50Yv67x1UA15/6h5OYXBa5wNURr/kqpes1qM1htFRs1dFHQIlpLFv+yXljl/Js1CaF1THiV71PTAhL8RWSMiGwRke0i8mgZ6S4VESMivqfRKaUakaoP9RwxbT5dn/rB7/ShFHJ38P9YbWvPIlv9HrFTFVUO/iISBLwOnAd0B64Wke5e0kUB9wBLq5qnUqpx+XjpHp/n9mTlUWj1/0tkqGUtbSSTV4supbZHG9WmQNT8BwHbjTE7jTEFwKeAt2epZ4G/AhXbTVkp1YD5F3wf/2pdpe6+aHvJ/Xst2Lg7+H8cNjGNutYPgQn+bQD3XZTTHMdcRKQ/0NYY831ZNxKRW0VkuYgsT09PD0DRlFKN2R+pWSVej7H8QW/LLp4rvLZRrePjTbV3+IqIBXgZeKC8tMaYt40xA4wxAxISEqq7aEqpRiSKPO4K/or9JpbvbA1jK8aqCETw3we4L3CR5DjmFAX0BOaLSCpwGjBTO32VUjUljAL+G/oCHWQ/TxTehE0HOgbkE1gGdBKRFBEJBa4CZjpPGmOOGmPijTHJxphk4HdgvDFmeQDyVkrVaxUf7ZNfaPV5bl3aUa/HHwn+lN6yi7sK72KerV+F82yIqhz8jTFFwJ3AbGAT8JkxZoOITBWR8VW9v1KqYSu02kh+9Hv+syTVr/SXvLGY/dknSh3fmZ7LFW8tKXW8h6QyIegnPrcOY7ZtUBVL23AEZIavMWYWMMvj2NM+0g4PRJ5KqYZAyDtpr8lPm72F609PLpVin0eg33jgGGe8+EupdGf//ddSxyLJ453QaWTSjOeLrqlaSRvYqFBd3kEpVSf4agBativLx5nyXRS0iFaSxcSChzhGZKXvA7qNo1JKBVY11aiT5QAPB89grS2FX219qieTekyDv1KqwekpO/kqdDJFWLij8F5MAEJdQ2v20eCvlKpFgW1LCaOAq4Pm8mXoMwiGywsmk2Z0zpA32uavlKpz9mTmYTC0i4vwK70FG6dZNvJs8Pt0sBxgvS2Zewr/zA7TpvyLy9A1MYrNB3MASGretEr3qms0+Cul6pyhf5vn+vmx87qWmTaMAv4d8neGBq3jmAnnroI7+cE2iKIqhrerBralRbMmbD6YQ5+kaO4Y3qFK96trNPgrpWqRW0O6jxagv/ywucw7PB/yHmdZ1vNM4QQ+sZ5NPmEBLJ/diK4tCA5qWK3kGvyVUjUmv9BK16d+dL3+edMh18/SJl8EAAAgAElEQVTO2H+yyPcMXk8Tg37ksqAFvFl0Ae9bzwtUMQF7B2+fpGgAejv+bkg0+CulakxG7snSBx1R3znT1x+9ZQfjgxZzc/APzLf24eWiywJYSifhnG4tWfLY2bSKrn87dZVHg79SqlKKrDaO5RcRGxHqV/r0nJOc9dd5pY4bR/Q/WWTz4y6G8ZbF/C3kLcKkiAXWXtxd+OdqXZ65IQZ+0OCvlKqgrYdyOPeVBbSJCWdf9gm2PnceocHlt4enHcnzerwiM2f/HfJ3RgWtZJ0tmRtOPkoWzfy/uIIa2rh+Tw2rB0OpBsgYw2/bMjB1ZH2BpY7lFpxr7hTZ7DX2xdszyM4rqPD9/HlXzcjl5ZA3GBW0knnWPlxWMKVaAz/Axf2qNky0rtPgr1QdN2PZXq57dylfr95XfmI/fLA4lZV7jlT6em8V4vxCK9e8s5Qbpy/zek2R1cb9n63xem7xjgyvx536yHa+DX2S8y1L+G/RSP5ceA8n8a+pqbJSXxzHwOTYas2jtmmzj1K1aMHWdF7+aStf3Ha6z6GEex3NJfuOlF7G2B8nCqzYjCEizP7rPnnmBsAe4AKlyGavv285mMPRE4Wk5+QT0zSU+Ej7sMuOT/zg89o7P17l81x/2cqM0GcpJJhrCp5guSl7zL/ynwZ/pWrR/Z+tJiO3gCN5hSREBX58OsCA537ieIE1oMHel7wCK32emeN6vXHqaJqGVi7MdJB9vBf6N/II4+KCqew0rQNSxg4JEXRsEcnsDYfo0zaGNXuzA3Lf+kabfZSqgvxCK0fzCqs1j6o29R8v8H/cvD88O0LLKt/O9OPM2XCwwnmMtizjm9CnsCFcW/B4wAI/wA1nJLt+vm1o+4Ddt77R4K+UF1nHC/zqYL3o9UX0mTqn3HTV6cf1B0h+9Huvo2kyvY2r92H9vqMkP/o9K3ZnYYxh8As/89myveVeZ/C9KvPVb//Orf9d4XcZAM6wrOe1kH+y07Ri7Mm/sN4ENkAbA/eP6kLnlpGc0TG+xLk7hnfgvYmNY3txDf5KediRnkv/Z3/ig8Wp5aZ1LvpVVQbDiQIr577yKyt2V2zzki9W2DuCN+w/Vurcs99t9Hld8qPfM8XR/g+wYFs6AD9tPIzVZjh07CQPf7mWPZklv1TEI9T3nDyb1T6aTnJOFvn3JhxiOcZrIa+xx7TkxoKHOUhcha73V5fEKObcN4zo8BBWPTWKVU+N4r6RnXng3C6c3bVlteRZ12jwV8rD7szjAMzfml4DuRUH0o0HjrH1UC7Pfb+p3Kt+25ZB8qPfs+1QTpnj0QutxU8vBUW2UpufT3f7gnMGdeOadmW3YX/xpujvL9rFAi+fy7XvLC23zOVJlgP8L3QyzcjjnsI7yaRmllRoHhFK84hQ7hnZiSBLAx/c70Y7fFWdk5pxHAOkxPu3nG+guYJgNQ6rv/ad30mIdOvgLSMvZ3D/cuU+zu/dmsvfWkJ6jr05Z/aGg2Qd9z223ub2Jka98iu7M71PtHLPB1PyvTuPb9x/jGe+9f0kUVkWbIy3LGZyyH8wwM2FD7LBJAc8H1WS1vwbCKvNsDfL+y/2f3/fzeLtZY+lrkuGT5vPiGnzA3a/2RsOsu1QBZpnHMEuELG/yGoj10vTx6LtmXy9er/Xa3x96ezKOM7wafNdgR9g2pytrNjte8y++718Bf5Xf94GFD+D/LY9o8SXxss/bQVg7D8X+synsqLI4/PQZ/hH6BvsMS24uGBqtWy52NAnbFWGBv8GYtqcLQx5aZ5r1qW7p75ezzUBeCyvr/703xWMemWB3+ndH/ytNuM1ePvrrk9W0XPybL8y9dZ8Y7MZ3lm4y6+8nM1VJa734/HllZ/twd2Z/4b9x0p8oWw9lFsts4t7yw7mhD1MP9nO04U3cHHBVHabxIDnA/Y2flWSBv8GYpGjZp+R4//ojuqWnVfAyz9txWqr/mUJ8gut1ZLPs99tpOfk2RVaZtjdD+v9G+ZY4GVRsz92ZdH+8Vl+LngGL8wqve69v5+I1WZKPBl4tuEH8qMNJ5/JwR/wZegULNi4ouAp/mMdjU3DUY3ST7seO3wsn9GvLGC/l9p+XTB55gb+OXcb8zYfrtZ83vttF12f+pGHv1gbkPuJONv8DV+uSAMgv9C/AOwur6D4icE5FyA7r4CvVqW5pbJH1UluyyKs3pvNsL/N44q3llQ4Tyfn04q/NfYOj8/io6V7fJ7PPB6YSkU7Oci/Q/7OjcGz+dw6lHNPvlQjs3bryLJIdYp2+NZjny3fy5ZDOXy0dHdtF8WrE47JRUWVrDYWWu0BN6ScHZSmOoYzfrkyjb9fYW8vzs4rICO3gI4tIiucr7fxHlsP5ZRa6+XnjYdKpevzzBxGdW/JtMv7MOC5n4uPT51D6ovjuPvT1SVGy2TkFjjun1viPmV1zJYl92QRqRnHOf+13yp1vS+Dnp9bpeuT5DD3BP2PS4IWUkQQDxXeyufW4YEpXAU19NU6/aXBX9VZA5//mZOFNjY9O8bva1bvzaZ1dBMuen0R+4/mV2lJA/fa4uVvLuGvl/biwr5taBIShM1muPk/y13nJ01fxmntYzl6opAvVqTx5xEdyfOYWXuyyMqho/k+83tj3o5Kl9XJr/6FGhBOPv0s2xls2cwQy1r6W7Zz0oTwnvU83i4aRzrNa7Q87oNXz+1ePf0K9U1Agr+IjAFeBYKAd4wxL3qcvx+4GSgC0oFJxpi6WV2tRxr6o2x2JZZNuOj1RUQ1CSYnv+xO2umLdjHl241sfnYMTUKCSpzzVTN85Mt1PPLlOgB2vDC2xLlfNh/mF7fmrZEv/1rq+j92ZbGljFFH7lsa1kdNOMkoywpGBK1mlGUFUXICqxHWmxReKbyUGdbhAZ20dVr7WH7f6d+EOOfvym3DOpAY3SRgZajPqhz8RSQIeB0YBaQBy0RkpjHGfUDwKmCAMSZPRG4HXgKurGreys5zxmVl5BdaCQu2uNq7fZn67UbO7BjHOd0qMgsy8N9S7yzcyd/nbGX15FGlzrkH/hM+1rX5l6OWnZ1XyHuLttImJty15ov7ZCdfM1QXbit7Api3zmdvzUT1lyFJ0ukv2+lr2U4/y3a6SyphUkSOCecH6yBm2QazwtaZHJrWWKkePa8rKfERPPj5Gtf/gysGJLnOa5NPsUDU/AcB240xOwFE5FPgQsAV/I0x7nu3/Q5cF4B865T3F+2i/ynN6dM2praLUiF7MvNYuD2dJ75az30jO3PPyE4s3JaOIAxuH1uqvf29Rbt4b9Euv5pTnL9oJwpLB+Cs4wX0f/Ynpl7Yg+tPT65wuZ2zYNPLGd104Kj3znD3vWTfXrATKF7wy1nusp6sJr7vfd36snywpP4+7IZRQG/ZyQDLVvpbttHXso0EsS8nccKEsta0533rGBbaerHE1qNGRu54+/e5bVgHAAanxNJ36k8A/PXS3vzmGA3Xr579flanQAT/NoD76k9pwOAy0t8E+F7cu55yznwM9LK5e7PySGoeXqJGXlBk45vV+wJSnz7/tYUcc9SQvlqVxp1nd2TCu38AMOnMFJ6+oHul7+385bxvxhou7pdU4tzyVPvj+tPfbOCyU5MICbKU27HrjaWKVblnvy85Y9VqM3y31vvkq8YijAJ6SCqdLWn0lF30seygq+wlROxf4jtsrVhg68MqW0dW2TqyxbSlqBa6D8v6/x/TtHizFxFhSKcE/nj8HFo00yYfpxr9FxOR64ABwDAf528FbgU45ZRTarBkddOWgzmM/od9ctKMW09jcHt7e+k/527jX/O207WciSv5hdYSa5V8sSKNS/u34YPFqVxyahJ7s/JcgR8gNTOPnPzidvY1adW3zrn7l1n3p2cTHR7Cmsnn+nWt+/DFy98seziktwCR/Oj3rp+/X3ugxLkPf9/NJ3/Y6zLO0UYNXRxH6WlJpZ9lG6eKvWYfIfYno2MmnLW29rxtG8dqW0eW2zpzpJq3TyxLn6Ro1qQ51hqqYO1HA39JgQj++4C2bq+THMdKEJGRwBPAMGOM12d1Y8zbwNsAAwYMqDPdmXkFRXR/ejavXNmnVA3Wl4+X7mFYlwTaxISXmS4956TPTTzcl2v404crWP20PTg6myyOnSi7Q7TrUz9yarviURUPfr6GjNyTvPjDZqb4WKPl7k9Xu36uzKzOb1bv479LdvPF7WeUOmezGZbvPkK7uNJtwEfLeS9bD+XQqUUkIlLicd/bjGZfjDFl9mncP2M1v+/MdL1ellr5rQ7rqliO0cWyl+6SSn/LNvpYdpIk9iYRmxG2mLZ8YR3KIltPNpp27DPxmDo0Hejf1w9g0Av2Yaf+bBqvfAtE8F8GdBKRFOxB/yrgGvcEItIPeAsYY4yp3hk/1WB/tn143mtzt/sV/LPzCnj8q3W0T4jglweG+0y3aHsG176zlLcmnMroHvbhZyt2Z/Hlyn08f1HPEmnLatz4ZfNhLB6/B6sce7R6rvtSXpDdcbh4vPnKPdm8MGsTj4/tVuY17u5xfHmcKLCW6FzLyD3Jyz9t5WPHRKKnzve/OWnupkPc9MFyXrmyD+d2T/Tah+CL53DNz2873Wfa/60KzB65dUULjtDLspNell2kyEE6y166WYpbaNNMPKtsHZluG80Gk8w6Wwq5Ndg5W1kLHx6B1WYIDw1isOOLYFjnBH6tkVVYG44qB39jTJGI3AnMxj7U8z1jzAYRmQosN8bMBP4GRAKfO2pee4wx46uat78Wbkvnn3O38emtp1dpyVabMeXWHu3p7H8fKWO1RYC1jsfXlbuPMLpHIjab4dL/szdjTB3fw+dTrTP7/Y4x4xsPlFzHfX/2CS5+Y7HXa8vbB9ZzLZi3F+z0Gvxv/mA5P286xI/3DiE6PARjoLXbU86fPlxBaFDx5+Q+4Qm8rzM/d9MhTu9QeijgjnT7F9J9M9YA3jcB98V9yOXy3UdYvCOzjNT1k2CjjWTQQ3bT3ZJKL9lFT0sqLcTebGczQpqJJ9Uk8m3hGawzKWyytSOjhpZMDiiBtrGlv6DemzjQr3WMVLGAtPkbY2YBszyOPe3288hA5FNZzg7MmWv2ERsRxrTZW/jqjjN8bpjtS2pmHrf8Zzm3DGnPgOTYCn2R5BdamTR9GZMv6EGXxCiMMXz4e8nRH1+VUfN0/8KZs6HsIYNpZQT4mWvK7sz09gv00o+bmbPxEB/fUtyP7xyTPnv9IdfCYO6d3d7WfC/PTR8s56HRXUod97buTWV5rmdfvxjiOUZny166iP1PN8seOsk+mjra6K1G2GaSWGjrxTpbCuttyWw0yeTRMNu728SEsy/7BEEWISgAQ54bk0Y1w/fLFfvYeiiHwzknycgtKDXZY/vhHJqEBJHU3Pej78+bDvPzpsPcfFYKX6xM49eHRtCsSfkf40pHrXPKzA18cutprNqb7WqvdobbI3nFTwqeIdjZ+XiiwEpmGU8UWccL+HxF+Vvv+XLoWOnumDfm28fEe5vi76yVg32N+qrytpn2tDlbq3xfp6qODqopzThOJ0mjiyWNzrKXLpJGZ8te4qR4klimiWKLrS2f2M5mu2nNRls7tpi25FM9G8HXRd/ceSZ7fCxlrsrWqIK/zRhXbd3qpYY78mX7yBpnDdZqszfzeAtI7/xmX2b3yreW8P3dQ7zm556D8ynDGcRz3UbZeOtY3X44lzfmb3e9dk5YKbKVXQt+4PM1tI6puVqe+5PEou1Vb1KZU80ToTyXXKhtkeTRy7KLDrKf9nKADrKfTpY0WkvxzNUcE85Wk8Rs6wC2mrZsMW3ZakuqsZ2u6hSPX5X4yDDiI31/2TVvGlLNBaq/GmTwtzka3S0ezTI2Y1w1P5vNcPuHK0iJj+DhMSVXFVyWmkV8ZBg3fbCMnenHOadrC595HTqWX2o2pzPX7LxCXvpxMw+P6YqzKJnHC/ho6W7auj1dHDiaz3+WpJZo2jnv1dIbZ/yxK6vclR6zjhfQwsfoIQV//nhljeYXSR5tJIM2kkFryaSlHKGVZJEk6Zwih0oE+TwTxk7Tit9t3dlqS2KzactWW1v2E0fZXf7Kmw3PjG5U2zJWVIMM/qf9ZS42A8ufLNnV8PvOLFdNwGaMa631TQeOkRJfvPqj59jxuWUsSXwkr7BEO/mnf+xxjdwBe5NJcnwE7R1bEu7KOM4TX63nabfRLt+tPcB3aw/wp2Hty3xfv/m5G1egNhVX/jC0IJsUOUg7y0G6yl6SJN0V8GOk5AYrViMcojn7TDxLbD3YaWvFepPCVlsSB2lep4ZV1ncRYQ0yvAVMg/x0Djum/L+zcGepzs8jjsXClu4qrnHN25LOvC2VHyb2wGfFI1Ae/d+6EsEf8LrOfIGXCURWa9mjFf45d1slS6gqwzmKpiVHSJCjtJAjtJBsWpBNC8mmpWTRTg67OlsBjpsw9poW7DPxrLB1Zp+JL/Eng2jdtETVCQ0y+Ds513/xJlAbfwB8v67kLNGL31hU7jXeRqXNWFb5jlpVOeHkkyQZtJXDJEk6yXKIDrKfJEknSTIIk5LzIoqMhQyiOWxi2GfiWWzryS6TSKpJJNW0ZL+J1+DupyfGduP5Wd5/R1NfHFdiJraLtuIETIMO/rUl1Y+NOIyXUfy+VpBUlRdOPq0lk0TJoq2k004OkSTptJV0kiSdeCk5R+K4o919s2nLz7b+7DKtOGDiOGxiSDcxZBGlwT0A/nl1P8b3ae0K/neO6Mi/5m33mnZwSiw70o/bZ7brUP6A0eBfS8qbbKXKF0YByXKQNpJBK8kijmMkShatHMG+tWQSLSW/iAtMEPtMPHtNC+bYTiXNtGCvSSDNJLDXtCCDZmj1MjB+um8oo15Z4Hp9+/AOdGoRyf2frWGQx65oD47uwoOju7Bx/zEW7yjZt1VPRufWOxr8a0lZ+6UqAEMsObSSTJIknVaS5ai9H6a9HCResokjB4uUrApmmigOmDjSTAv+sHXloIljv4nlgIljn4nnAHH1vuY+oF1zlu8O/LpDF/drw5UD23LV2/b5Ghf2bc03q0tOCuyQEMGO9OJO7ISoMK/Laq9/ZjSRHh2u43q1omebaC7pX3KJlG6tiheK6966Gd1b21/fMiSFfy/c1eA3LaotGvxVrWhKPi3lCC3IpqUcIUGOkCQZrmaZJMko0ZEKcNKEsN/EstO0ZqWtE4dNDDtNK9JMAgdMLJlEU1hD/6VvOiuFNjHh9Gkb7VqSo6qGdIpn4bbyR3RNnzSo3O0a+58Sw8o9FVuVVSgZiHu1iS4R/P91TT9GdGnBOwt3uWZ1//H4OaQ8NsvzVoR6mT3vbQetb/58JslxEV7Lc3bXlvx7oX0+jdb+A0+DvwogQzPySJQsmnGcOMmhpWSRKEdoKVkkSQYJZBMvR2kmpZu9ck0T9piW7DKtWGDrzT4Tz34TT5qJ54CJI4soarpJpmtilNehs86F6cpb9vmvl/Zybf1YnutPT/Yr+PvzCdwypD23f1TxOQ3R4SFsfe48Zizbw3WntaPfKc0ptNp4+Iu1nNUxnoiwYO4Z2YnrTjuF4yetPte5Cgkqfdzb0iFlbX7krV9MBY4Gf1WuJpwkSdJpLZm0crSlJ5JFc8khWo7TjOPESg7NySFUSs+gLTRBHKI5B0wsG0wyGbZoDpnmHDSxHCaGQ6Y56SaGYzSlrrW3/3jvUO+jThzKWy4iEFtslrqnl1uO692qxN4E/oTNiNAgjnuZ8RwabGGCY3c155LgCx4eUSJNXGQYcZElr7tvZGfXE4G3L4XKNt9orb96aPBv5IIpogXZjo7SLFo7JiclSDaJcoREySKRLILc2tZtRkgnmiwTxTEi2GtasNrWkSNEkWmiOGRiOUoEWY6fM4mqF5OXHj2vKy/+sLlC15Q3gdR9hul3d53F+a/95jOt+zIf8x4czohp88vN/4FRnfn7T1tJ9rJHgrvbh3fg/lGd6fRE8SZ6ngG6Q4tIz8sq5Nah7ckvsvKeY+kTT02Cgyp1X2Mq/8WhfNPg34AFU0RLjtBGMlzBvfhvey0+gaOlOk2Pmaakm2gOmliWmB7stSWwyySy38RxwMRxiOa1sm2fL4+d15W/VDBoe1OZCqaI8N+bBhERFswlHstop744jq9Wpble92wTzUc3D2banC2s8tIe7/6vkBLvvR28spoEB5XaJtPz/Tr3v60sEXhkTFce8VguBaBTi0iidZ2dOqXu/Aarcgk2osgjTnKI5RhxcoxoOU4Mua529ZZyhGiO00zySCC7RI0d7NvyHTSxHDSxbLGdwkFiOeB47fz7KFWrAVbG3ed0Iie/kPcXpVb42jE9EwMS/AFaRIW5Zoh7mnhGMtMXp5Y6PqRTAgBz7huKRYQ7PlrhGtHiWWM9s2N8iQX7ynLH8A60ignnqa/Xlzjufk9vTSLegm+wow3+2Yt6uu7XolkYOen2uSXNmgRXeR0cb01gp7ePY8nOTLqUs+VoWUS06ac6aPCvRVf2TeDX1ZuJlRzi5BixHLO3nUsO8Rwj3rGkQAzHiZI8ojlOsHjvYDxuwjhoYjlkmrONNuTamnKIGPY5RsLsM/EcMs2rbaemh8d04aUft5Sb7pNbTqNTy8hSm7sMSo5ld9ZxH1eVLZDt6n8e0ZHJMzeUOOYchTMwOdZr8Hfq3NIe4ObcV7xFtTNQX9yvjeuYr34C51Hn9oTOBQedwfr1a/qz5VCOa82aUd1blrj+1av60rNNNB0SIpnlmHXeqUUku7PyuKB36xJ5NAmx8NHNp/HJH3t4de62gDSreHtbVw1qy5KdDW8DnYZAg3/AGKI44bb+yxFayhFiJYcYcomR48RIruNn+99NNhfibY8NmxGyiCLD2DtGd5PIUVsEx2jKERNFpmlGFlFkmSiyieKoiSCHcKqzs/SXB4Zx9t9/9Xm+Z2v/lhc+rX2sz/0IqtLU8e4NA7jpg+WkxEewK6NyXyJQciXY8JCSbdQRYZVrs4aS/zLndG3hdVTP2V1bsOGZ0T5rueN6t2IcrQBY8/S5hIcGcehYPm/9upPLTm3r9fPr2CKSn+4fVur4xf2SSIxuws1DUnh17raA7IJVVud3ebvflUXb+6uHBv9yFQd193Hp9oW9jpAg2bTE/tpzXDrYx6YfIZJsE8lRIkg1iWTbIjinfxcWZFiYt7uATNPMEdCbccREcowILJYgimz+/693dvwB/PbICM7667yAfQJgD8z3juzEsM4JXreI9LfJwFcQEIEzOsT7vO7Jcd0Y1jmBhKgw+k79qdS1bZrbt5AMDbJgkeKtNH2Zfe9QRv9jQYljQRbhrI72Mlw1sC1/crSBe5Z5bK+SC/eVxVsxbjgjmfF929D/2ZLvIzjI4vfucs7287axTVn3zGi/y9PvFPvQSucy5c5/t0CsgFmdLTOJzZqQnnNSl2gOoEYf/JuS71oSoK1jJmmKHCBRslyBPlxK11SPmzAOmeYcpjlrTXsO2Zpz2DiGLTqGLx42MeT6qJGnXjKOVT9u5pOdO7yW67rBp/CfJbu9nvPGfXJOkxD/aqi+xrB7IyLcO7Kzz/Onty+9964v3mpy7coZrXJqu+Z0almy3dg5A7RZkxCOFxSvizS8Swt+cSzD/fo1/emdFE3ziFDXxKgf7hlSog36f3ecwQ/rDnDdae1oEhJUYjvKEuUGn+d8cY3gcfsvICLERoT6fY+YpiFk5xWWn9APPVpHs+3581ydv01Dg3n6/O6cXcaeFf7y9r0+tFMCbWLCuWN45TuTReDdiQNYtD2DuDI2blEV04iCv6ENGfSx7KC3ZSd9ZCfdLLtLrbduM8J+4kgzCawxHThsKw7khykO8McJ95GP/+45pxP/N9978J9yQY8yg/8fj5/DoBdKb6sIxcE1LiK0RBOLr3Hd7lo2C/O6lWN5LBZh6ePnMNhRpo1TR3P7hyv51bGX73k9E107ojnbtJ02Pzum3C8sb08Mj57XjduHdyS6aQj7jxZPGnMfMtmxRWSpDb/dvyjB/jk9Ma47viQ2sweciNDK/7pUpV9i7v3DyMj1vXWnN2U1lXiO+pl0VkplilWKt3+j5hGhLHr07Crfu0VUEy7ul1R+QuW3Bh/8m5LPGMsf3BL8Pd0s9iWTT5pgNplTmGUdzB7TgkOmOQeIY68toUaHMTprmd4mEXnuQuapWbjvYXPOmZGev4sfTBrEvxfuJNhi4ZL+bfjb7NIdtHeO6MhT3xR3eH5x2+m08zH93lPLZsUdGE1Dg7l8QJIr+P/fdae6zkWHh/DhTYO57t2lQMknlb9f3ocHPi/eH8HJfZ/k/7u2v2vTbmcN2lXBLuNj69QikutOa1finsfyi8qdqDVlfA8GpcQxMLl5mem8ccbgsrK4dWh7drrthezJPqGqcjVeHSWjfGlwwf9kkb1m25oMJgX/wFVB84iUfLbZ2vBM4QSW27qw2ZxSY2vAVMb1p7crN02ZActV6yuZZkByLAPcVlN0D/7v3ziQG99fxukd4ujVJpp1+47yz6v7lUhfUc6APK53q1LnzurkvX3/0lOTMMCDbl8Ab153Ku0Tioefnter9P1K5Ov2s/t3qGfHZ6eWUazYfaTUk4inpqHBXHZqJWudpVt9Snl8bLfK3duf7LWzVPlQdyNgJRVaDTcFfc/DwZ8RQhFf287k46KzWWE61/gs0yfGdqN3UjRXOlZJ9GXFkyOxGZjy7Qa+X3vAa8B94eJePP5V8RoxocEWPvvT6V739HX+vrsHPm8bWTsDwzvXD2BElxal2rPLmzXqzce3DCYpxn5dV0e7+shuFWtP9gyUY3qW38HaPiGCU2Kb8uS47rzz207X8Y5lzFp9a8Kp/LYto8QTi1KNRYML/kFZ23kq5CPmW/vwZNEk0kxCrZXllqHtsdkMImXXwDwf6b3VEq8ZfAovzNpErtuGL4NSir8k4iKLOxCbNw0lLNjC446dktJzTtL/FN9NFkmx3vsvKlNrdB+x06llFBunjqapj5wW+8UAAAvdSURBVLby5LimXoOze7aewy19aRIS5Fp/5t8L7cH//RsHljnEMD4yjIvcxt9XB19NcNXttPaxRIUFu0YsKeWpwQV/E9uBawseY70tpdpmqi557Gwmf7OBQSmxZW4VCfa2+11/GcfJIitdnvyxzLSXn5rE92sP0Nex0mHftjGs3uvfsrz93IJ7aLCFLc+dB9g3n/92zX7G923t130CzVfgB5j/0Aif55wqM/7c1c5e4SsDr3iwT82WJi4yrEJDQKuiRVRYuaubqrqn7q+2VUEGYZGtV0ACf4eE4o7Oh0Z3cf3cKjqct68fwM1D2vt9rzA/FrUa7mh6cY5O+eDGQRUorXfOp4NArhXT3zFWvKLDHiujrI5tX4w2dNeoP54Yyaqnz63tYqgKang1/2q6b4/WzcpP5PDwmC5c2LfqzQmetV5nUPv8ttP9vsd1g09hRJcEkpr7335/Sf82rNt3lNYx3puD/nvTYDIrOPSwIqIdAT8iNIgvKvBenYZ1TmDhtgy/RylVJ39G+yhVGwJS8xeRMSKyRUS2i8ijXs6HicgMx/mlIpIciHy9qWit75cHSk999ybY4v9HdcfwjrTxEjjXTalY7cj5TmI8Oms9x6mXRUR8Bv6xjlEz8R59DhPPSGbb8+eREOV9eGFEWDCnODqDh3SK57Wr+/ldHn+M7NaCl6/ow8qnR1UqgN90VgrLnxwZ8JUxK8OfIaiqbPogVz2qHPxFJAh4HTgP6A5cLSKeM2ZuAo4YYzoCrwB/rWq+vlT0/0n7hEif0/WdnYXj+7TmjA5lz2B99sIe5eYV1aRiTRhhjiGIg1NKjv5x/4Jb+PAIfn1oeIXu63TX2R1ZO+XcUsFfREpNBPLlvzcN5oI+ge1PEBEu6Z/kV1OZr+s931Nt0d2oVF0ViJr/IGC7MWanMaYA+BS40CPNhcAHjp+/AM6Rqqz0VAbPWkJ0GW3Go3vYV0V01tJvG9aB7m61ameQvevsjj4nXQU7jjt3PirP/aM68787zvArbURYMLPvHco/rvRds24b27TSzRsWi9Csgl9IqmLG9mxF91bNqrxWfmOmT03VIxDBvw2w1+11muOY1zTGmCLgKFCqKi0it4rIchFZnp6eXrnSeAT/Vl42jfaSL2AfCz/rniEVym7uA8P49/UDAPviYx9MKruT9u5zOpU57NJTl8QowkPtNeCrB50C+O48vqiWRvQo35pHhDLrniF1ov9BKXd1qsPXGPM28DbAgAEDKvW87PmY7fmA0T4hgp3p3pf8rUyG7eIiXL/YFRn9UxmPj+3GQ2O6+JyR+sqVfXnlyr7VWgalakqkY6XR1tFVX0dLlRaImv8+oK3b6yTHMa9pRCQYiAaqZYeH8jqHUuIi+L9r+wPFY699PVV6fnG8dFnvCs9WDSSLRcpsBxeRKq2brlRd0jsphteu7sezF/Ws7aI0SIEI/suATiKSIiKhwFXATI80M4EbHD9fBvxiqmkwtudN7zq7IwBTL+zBsM4J/O3yPr6vLadEVwxoyzs3DKxiCZVS/rqgT+uA7DWgSqvyp2qMKRKRO4HZQBDwnjFmg4hMBZYbY2YC7wL/FZHtQBb2L4hq4Tk2fmyvVq7JSNc7OmU9Y3yUY8VIXzs16XgNpVRDE5CvVGPMLGCWx7Gn3X7OBy4PRF7lsYiUuQm3O2cLyS1D2xMWHMQ1jg5V1/nqKKBSStUBDW55h9iIUP54YmSZaTybd8KCg7hlaPtSW+hpjV8p1VA12Ma0tyacSqKPpXqbhdvftr9L+eoTgFKqoWmwwX90D99rwJ/VMZ5Xr+rrM82Ll/Ri66FcFmyzzzXQJwClVEPTYIN/WUSkzIXXrnK0/d80fRnbD+fSpJLLDCilVF3VKIO/v165qi+Lt2e6FjFTSqmGosF1+AZSsyYhfm0hqJRS9Y0Gf6WUaoQ0+CulVCOkwV8ppRohDf5KKdUIafBXSqlGSIO/Uko1Qhr8lVKqEdLgr5RSjZAGf6WUaoQ0+CulVCOkwV8ppRohDf5KKdUIafBXSqlGSIO/Uko1Qhr8lVKqEdLgr5RSjZAGf6WUaoQ0+CulVCOkwV8ppRoh3cBdqVoy5YLuDEyJre1iqEZKg79StWTimSm1XQTViFWp2UdEYkXkJxHZ5vi7uZc0fUVkiYhsEJG1InJlVfJUSilVdVVt838UmGuM6QTMdbz2lAdcb4zpAYwB/iEiMVXMVymlVBVUNfhfyP+3d24hVlZRHP/9ES+RVt4yUUkFIXwIk0GMRMS8lEX1ICEETRcQ6qXoIZSBoLfqISoIRCywqLQsaTCixhR6StO8TZnOZEbZ5EyZVi9dVw97HTsM51PrjH3n7LN+sJm1197zfesPe9bss7/v7A0b3d4I3DG4g5kdNbMet78F+oGJdd43CIIgqIN6k/8kM+tz+ztg0rk6S5oHjAC+KGhfLWmPpD0DAwN1hhYEQRAUcd4HvpK2A1fVaOqorpiZSbJzXGcy8DLQbmZ/1epjZuuB9QBtbW2F1wqCIAjq47zJ38yWFLVJOilpspn1eXLvL+h3GfAO0GFmH/3naIMgCIIhod5ln06g3e124O3BHSSNALYCL5nZljrvFwRBEAwB9Sb/J4ClknqAJV5HUpukDd7nTmAhcI+k/V7m1HnfIAiCoA5k1phL65IGgK/quMQE4PshCqfRyFkbhL5mJ2d9zaDtajM77xuVDZv860XSHjNrKzuOi0HO2iD0NTs568tJW2zsFgRB0IJE8g+CIGhBck7+68sO4CKSszYIfc1Ozvqy0Zbtmn8QBEFQTM4z/yAIgqCASP5BEAQtSHbJX9JNko5I6pVUa4vphkTSi5L6JXVX+Wqel6DEc67xoKS5Vb/T7v17JLXXutf/jaRpknZK+szPdXjI/bnoGyVpt6QDru9x98+QtMt1bPZvuyNppNd7vX161bXWuv+IpOXlKKqNpGGS9kna5vVs9Ek6LumQfwl1j/uyGJ+FmFk2BRhG2jF0Jmn30APA7LLjusDYFwJzge4q31PAGrfXAE+6vQJ4FxAwH9jl/nHAMf851u2xDaBtMjDX7THAUWB2RvoEjHZ7OLDL434dWOX+dcADbj8IrHN7FbDZ7dk+ZkcCM3wsDytbX5XOR4BXgW1ez0YfcByYMMiXxfgsKrnN/OcBvWZ2zMx+AzaRzhxoeMzsQ+DUIHfReQm3k/ZKMksb5V3hG+stB7rM7JSZ/Qh0kQ7QKRUz6zOzT9z+GTgMTCEffWZmv3h1uBcDFgOV/awG66vo3gLcKEnu32Rmv5rZl0AvaUyXjqSpwC3ABq+LjPQVkMX4LCK35D8F+Lqq/o37mpWi8xKKdDa8fl8CuI40O85Gny+J7CftbNtFmtWeNrM/vEt1rGd1ePsZYDwNrA94BngUqGzHPp689BnwvqS9kla7L5vxWYs4wL1JMDv3eQnNgKTRwJvAw2b2U5oMJppdn5n9CcxROqJ0K3BNySENGZJuBfrNbK+kRWXHc5FYYGYnJF0JdEn6vLqx2cdnLXKb+Z8AplXVp7qvWTnpHycrh+FUzkso0tmw+iUNJyX+V8zsLXdno6+CmZ0GdgLXk5YDKhOs6ljP6vD2y4EfaFx9NwC3STpOWkpdDDxLPvowsxP+s5/0z3seGY7PanJL/h8Ds/wthBGkh02dJcdUD0XnJXQCd/tbB/OBM/7x9D1gmaSx/mbCMveViq/3vgAcNrOnq5py0TfRZ/xIugRYSnqusRNY6d0G66voXgnssPTEsBNY5W/LzABmAbv/HxXFmNlaM5tqZtNJf1M7zOwuMtEn6VJJYyo2aVx1k8n4LKTsJ85DXUhP4o+S1lw7yo7nX8T9GtAH/E5aK7yftE76AdADbAfGeV8Bz7vGQ0Bb1XXuIz1I6wXuLVuXx7SAtKZ6ENjvZUVG+q4F9rm+buAx988kJbde4A1gpPtHeb3X22dWXavDdR8Bbi5bWw2ti/jnbZ8s9LmOA14+reSNXMZnUYntHYIgCFqQ3JZ9giAIggsgkn8QBEELEsk/CIKgBYnkHwRB0IJE8g+CIGhBIvkHQRC0IJH8gyAIWpC/AUQjl60KodC9AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe5af3d2550>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# ranked predictions plot\n",
    "pred_frame = local_glm2.predict(practical_sample).cbind(practical_sample)\\\n",
    "                       .as_data_frame()[['predict', yhat]]\n",
    "\n",
    "pred_frame.columns = ['Surrogate Preds.', 'ML Preds.']\n",
    "pred_frame.sort_values(by='ML Preds.', inplace=True)\n",
    "pred_frame.reset_index(inplace=True, drop=True)\n",
    "_ = pred_frame.plot(title='Ranked Predictions Plot')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both the R<sup>2</sup> and ranked predictions plot show the linear model is a good fit in the practical, approximately local sample. This means the regression coefficients are likely a very accurate representation of the behavior of the nonlinear model in this region."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 7. Generate reason codes using a practical sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Create explanations (or 'reason codes') for a row in the local set\n",
    "Reason codes are generated for the model based on the practical sample, just as they were for the model based on the perturbed sample. Again, the woman's value for `PAY_0` is the most important local contributor to her GBM prediction. As seen in other attempts to explain this prediction, `PAY_0` followed by other payment variables, marital status, and her age play a role in the model decision. Also, for better local accuracy and explainability, LIME contributions are scaled such that contributions for each prediction plus the LIME intercept term sum to the GBM's predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gbm prediction progress: |████████████████████████████████████████████████| 100%\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAF8CAYAAAAjJWv0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvpW3flQAAIABJREFUeJzt3XmYXGWZ/vHvTYCwhwBRNOyLIIgKtgiCG8imIzgIEhQFh9+gjigjKoOigjguoLigiKKAgKOgCBpFRURAHdk67OsQg2wihk0QEAh5fn+8b5PqSi91Tp3qOp1zf66rrlSdU/XUczrdT516z7soIjAzs2ZYqt8JmJnZxHHRNzNrEBd9M7MGcdE3M2sQF30zswZx0TczaxAXfbMaknSxpP/X7zxsyeOib7Ug6c+SnpD0D0l/lfRdSSv1O68iJG0t6ReSHpb0oKQrJL2r33mZtXLRtzp5U0SsBLwU2BL4aJ/z6ZikbYHfApcAGwGrA+8FdutnXmbtXPStdiLir8D5pOIPgKSpkr4o6U5J90n6pqTl877pkn4uab6kh/L9tVpee4CkeZIelXS7pLfn7UtJ+rikOyT9TdLpkqblfetJCkn75/e8X9IRY6T9BeC0iDgmIu6PZE5EvLUlj3+XNDd/C5gt6fkt+3aSdIukv0v6OqDW4JL+TdLN+fjOl7Ru3i5JX875PyLpekkv6uLHb0s4F32rnVywdwPmtmz+PPAC0gfBRsBM4JN531LAqcC6wDrAE8DXc6wVgeOB3SJiZeCVwDX5dQfk2+uADYCVhl7XYntgE2BH4JOSXjhCvisA2wJnj3FMOwCfA94KPA+4Azgz71sDOAf4OLAG8Cdgu5bX7gF8DNgTmAH8HvhB3r0z8Or8s5mW4z8wWh5mRIRvvvX9BvwZ+AfwKBDAhcCqeZ+Ax4ANW56/LXD7KLFeCjyU768IPAy8BVi+7XkXAv/R8ngT4GlgaWC9nMdaLfuvAGaN8H4z83M3HeP4TgaObXm8Un6v9YB3Ape17BNwN/D/8uNfAge27F8KeJz0IbcD8H/ANsBS/f5/9K3+N5/pW528OdLZ+GuBTUlnvZDOblcA5uSLpA8Dv8rbkbSCpG/lZppHgN8Bq0qaEhGPAfsA7wHulXSepE1z3OeTzriH3EEq+M9t2fbXlvuPk4p1u4eAhaQz+NEMe6+I+AfpjHxm3ndXy75ofUwq7l9tOfYHSR8MMyPit6RvJycAf5N0kqRVxsjDGs5F32onIi4Bvgt8MW+6n9Rks3lErJpv0yJd9AX4EOks/RURsQqpuQNyu3hEnB8RO5GK8i3At/P+v5AK6pB1gAXAfQXzfRy4lPRtYjTD3is3O60O3APcC6zdsk+tj0kfAO9uOfZVI2L5iPhjfv/jI+JlwGakZp6PFMnfmsVF3+rqK8BOkl4SEQtJhfrLkp4DIGmmpF3yc1cmfSg8LGk14MihIJKeK2mPXGSfJDUhLcy7fwB8UNL6uXvoZ4GzImJBiXwPAw6Q9BFJq+f3fomkM1ve612SXippan6vyyPiz8B5wOaS9pS0NPABYM2W2N8EPipp8xx3mqS98/2XS3qFpGVITWD/bDk+s8W46FstRcR84HQWXaz9L9KF3ctyE85vSGf3kD4glid9I7iM1PQzZCngUNKZ9oPAa0hdKQFOAc4gNQfdTiqY7y+Z7x9J7es7APMkPQicBPwi7/8N8Angx6Qz+w2BWXnf/cDepIvVDwAbA//bEvtc4BjgzHzsN7CoK+gqpA/Eh0jNRw+QehKZjUip+dDMzJrAZ/pmZg3iom9m1iAu+mZmDeKib2bWIEv3O4F2a6yxRqy33nr9TsPMbFKZM2fO/RExY7zn1a7or7feegwODvY7DTOzSUXSHeM/y807ZmaN4qJvZtYgLvpmZg3iom9m1iAu+mZmDeKib2bWIB0VfUm7Sro1r+95+Aj7D5V0k6TrJF04tH5n3veMpGvybXaVyZuZWTHj9tOXNIW0Ks9OpCXcrpQ0OyJuanna1cBARDwu6b3AsaTVigCeiIiXYmZmfdfJmf7WwNyImBcRT5EWc96j9QkRcVFePQjSfOZrVZum2eQnjX8z67VOiv5Mhq/XeXfeNpoDSQs5D1lO0qCkyyS9eaQXSDooP2dw/vz5HaRkZmZlVDoNg6T9gAHS6kRD1o2IeyRtAPxW0vUR8afW10XESaRVhhgYGPCqLmZmPdLJmf49DF+kea28bRhJrweOAHaPiCeHtkfEPfnfecDFwJZd5GtmZl3opOhfCWycF49elrSu57BeOJK2BL5FKvh/a9k+PS8CjaQ1gO2A1gvAZmY2gcZt3omIBZIOBs4HpgCnRMSNko4GBiNiNmkh5pWAHyldjbozInYHXgh8S9JC0gfM59t6/ZiZ2QSq3cLoAwMD4amVbUnUSe+cmv052iQiaU5EDIz3PI/INTNrEBd9M7MGcdE3M2sQF30zswZx0TczaxAXfTOzBnHRNzNrEBd9M7MGcdE3M2sQF30zswZx0TczaxAXfTOzBnHRNzNrEBd9M7MGcdE3M2sQF30zswZx0TczaxAXfTOzBnHRNzNrEBd9M7MGcdE3M2sQF30zswZx0TczaxAXfTOzBnHRNzNrEBd9M7MGcdE3M2sQF30zswZx0TczaxAXfTOzBumo6EvaVdKtkuZKOnyE/YdKuknSdZIulLRuy779Jd2Wb/tXmbyZmRUzbtGXNAU4AdgN2AzYV9JmbU+7GhiIiBcDZwPH5teuBhwJvALYGjhS0vTq0jczsyI6OdPfGpgbEfMi4ingTGCP1idExEUR8Xh+eBmwVr6/C3BBRDwYEQ8BFwC7VpO6mZkV1UnRnwnc1fL47rxtNAcCvyzyWkkHSRqUNDh//vwOUjIzszIqvZAraT9gAPhCkddFxEkRMRARAzNmzKgyJTMza9FJ0b8HWLvl8Vp52zCSXg8cAeweEU8Wea2ZmU2MTor+lcDGktaXtCwwC5jd+gRJWwLfIhX8v7XsOh/YWdL0fAF357zNzMz6YOnxnhARCyQdTCrWU4BTIuJGSUcDgxExm9ScsxLwI0kAd0bE7hHxoKRPkz44AI6OiAd7ciRmZjYuRUS/cxhmYGAgBgcH+52GWeXS+dDYavbnaJOIpDkRMTDe8zwi18ysQVz0zcwaxEXfzKxBXPTNzBrERd/MrEFc9M3MGsRF38ysQVz0zcwaxEXfzKxBXPTNzBrERd/MrEFc9M3MGsRF38ysQVz0zcwaxEXfzKxBXPTNzBrERd/MrEFc9M3MGsRF38ysQVz0zcwaxEXfzKxBXPTNzBrERd/MrEFc9M3MGsRF38ysQVz0zcwaxEXfzKxBXPTNzBrERd/MrEFc9M3MGqSjoi9pV0m3Spor6fAR9r9a0lWSFkjaq23fM5KuybfZVSVuZmbFLT3eEyRNAU4AdgLuBq6UNDsibmp52p3AAcCHRwjxRES8tIJczcysS+MWfWBrYG5EzAOQdCawB/Bs0Y+IP+d9C3uQo5mZVaST5p2ZwF0tj+/O2zq1nKRBSZdJevNIT5B0UH7O4Pz58wuENjOzIibiQu66ETEAvA34iqQN258QESdFxEBEDMyYMWMCUjIza6ZOiv49wNotj9fK2zoSEffkf+cBFwNbFsjPzMwq1EnRvxLYWNL6kpYFZgEd9cKRNF3S1Hx/DWA7Wq4FmJnZxBq36EfEAuBg4HzgZuCHEXGjpKMl7Q4g6eWS7gb2Br4l6cb88hcCg5KuBS4CPt/W68fMzCaQIqLfOQwzMDAQg4OD/U7DrHLS+M+p2Z+jTSKS5uTrp2PyiFwzswZx0TczaxAXfTOzBnHRNzNrEBd9M7MGcdE3M2sQF30zswZx0TczaxAXfTOzBnHRNzNrEBd9M7MGcdE3M2sQF30zswZx0TczaxAXfTOzBnHRNzNrEBd9M7MGcdE3M2sQF30zswZx0TczaxAXfTOzBnHRNzNrEBd9M7MGcdE3M2sQF30zswZx0TczaxAXfTOzBnHRNzNrEBd9M7MG6ajoS9pV0q2S5ko6fIT9r5Z0laQFkvZq27e/pNvybf+qEjczs+LGLfqSpgAnALsBmwH7Stqs7Wl3AgcA32977WrAkcArgK2BIyVN7z5tMzMro5Mz/a2BuRExLyKeAs4E9mh9QkT8OSKuAxa2vXYX4IKIeDAiHgIuAHatIG8zMyuhk6I/E7ir5fHdeVsnOnqtpIMkDUoanD9/foehzcysqFpcyI2IkyJiICIGZsyY0e90zMyWWJ0U/XuAtVser5W3daKb15qZWcU6KfpXAhtLWl/SssAsYHaH8c8HdpY0PV/A3TlvMzOzPhi36EfEAuBgUrG+GfhhRNwo6WhJuwNIermku4G9gW9JujG/9kHg06QPjiuBo/M2MzPrA0VEv3MYZmBgIAYHB/udhlnlpPGfU7M/R5tEJM2JiIHxnleLC7lmZjYxXPTNzBrERd/MrEFc9M3MGsRF38ysQVz0zcwaxEXfzKxBXPTNzBrERd/MrEFc9M3MGsRF38ysQVz0zcwaxEXfzKxBXPTNzBrERd/MrEFc9M3MGsRF38ysQVz0zcwaxEXfzKxBXPTNzBrERd/MrEFc9M3MGsRF38ysQVz0zcwaxEXfzKxBXPTNzBrERd/MrEFc9M3MGsRF38ysQVz0zcwapKOiL2lXSbdKmivp8BH2T5V0Vt5/uaT18vb1JD0h6Zp8+2a16ZuZWRFLj/cESVOAE4CdgLuBKyXNjoibWp52IPBQRGwkaRZwDLBP3veniHhpxXmbmVkJnZzpbw3MjYh5EfEUcCawR9tz9gBOy/fPBnaUpOrSNDOzKnRS9GcCd7U8vjtvG/E5EbEA+Duwet63vqSrJV0i6VUjvYGkgyQNShqcP39+oQMwM7PO9fpC7r3AOhGxJXAo8H1Jq7Q/KSJOioiBiBiYMWNGj1Mys8lE6uxmnemk6N8DrN3yeK28bcTnSFoamAY8EBFPRsQDABExB/gT8IJukzYzs3I6KfpXAhtLWl/SssAsYHbbc2YD++f7ewG/jYiQNCNfCEbSBsDGwLxqUjczs6LG7b0TEQskHQycD0wBTomIGyUdDQxGxGzgZOAMSXOBB0kfDACvBo6W9DSwEHhPRDzYiwMxM7PxKSL6ncMwAwMDMTg42O80zCrXSbtzzf4ca6HT9vqm/+wkzYmIgfGe5xG5ZmYN4qJvZtYgLvpmZg0y7oXcOnCbnplZNXymb2bWIC76ZmYNMimad8xscnGTbH35TN/MrEF8pm82CflM2srymb6ZWYO46JuZNYiLvplZg7jom5k1iIu+mVmDuOibmTWIi76ZWYO46JuZNYiLvplZg7jom5k1iIu+mVmDuOibmTWIi76ZWYO46JuZNYinVjYbRZOmL27SsTadz/TNzBrERd/MrEFc9M3MGsRF38ysQXwh1/qqkwuIvnhoVp2OzvQl7SrpVklzJR0+wv6pks7K+y+XtF7Lvo/m7bdK2qW61M3MrKhxi76kKcAJwG7AZsC+kjZre9qBwEMRsRHwZeCY/NrNgFnA5sCuwDdyPDMz64NOzvS3BuZGxLyIeAo4E9ij7Tl7AKfl+2cDO0pS3n5mRDwZEbcDc3O8JYrU2c25mVm/ddKmPxO4q+Xx3cArRntORCyQ9Hdg9bz9srbXzmx/A0kHAQcBrLPOOoslUHWbrj7VWZWLIzt74zq3OTfpZ1fn3KqOV+fcqo5X99/hKk1EbrW4kBsRJwEnAQwMDPT8J92P/8wlhX92NtnV+Xd4InLrpHnnHmDtlsdr5W0jPkfS0sA04IEOX2tmZhOkk6J/JbCxpPUlLUu6MDu77Tmzgf3z/b2A30ZE5O2zcu+e9YGNgSuqSd3MzIoat3knt9EfDJwPTAFOiYgbJR0NDEbEbOBk4AxJc4EHSR8M5Of9ELgJWAC8LyKe6dGxWMPV+Wu7WV0oanYVcmBgIAYHB/udhpnZpCJpTkQMjPc8T8NgZtYgLvpmZg3iom9m1iAu+mZmDeKib2bWIC76ZmYN4qJvZtYgLvpmZg1Su8FZkuYDd3Tw1DWA+yt86zrHq3NuVcerc251j1fn3KqOV+fcqo7Xaax1I2LGeE+qXdHvlKTBTkafLQnx6pxb1fHqnFvd49U5t6rj1Tm3quNVnZubd8zMGsRF38ysQSZz0T+pQfHqnFvV8eqcW93j1Tm3quPVObeq41Wa26Rt0zczs+Im85m+mZkV5KJvZtYgLvpmE0jS6hXHe5OkSv6Oe5DbFhXHq+xYm8w/QGsMSc/vdw7AZZJ+JOkNklRBvH2A2yQdK2nTmuX2DUlXSPoPSdMqiFflsU4qkp5TVazaF/3W/1xJU9v2bdOD9yt8pVzSFEnvlvRpSdu17ft4iXgrSDpM0kckLSfpAEmz8y/7SiXivbjl/jKSPp7jfVbSCgVjHSxpjXx/I0m/k/SwpMvLntlJOkfSGyfgLO6yIk+W9LP8cxrxVjKHF5B6Y7yDVMA+K+kFJWMREfsBWwJ/Ar4r6VJJB0lauQa5vQp4O7A2MEfS9yXt1EW8Ko91RJJ+WeI1a0o6UdIJklaXdJSk6yX9UNLzSsRbre22OnCFpOmSVisab7H4de+9I+mqiNiq/f5IjwvEHO0HJ+DaiFirYLzvACsAV5D+YC6JiEPL5pgXk78LWB7YBLgZOAvYHVgzIt5RMF7rz/A4YHXgVODNwOoR8c4CsW6MiM3z/fOA70TEuZJeC3wmIrYbM8DIMV8PvAvYBvgRcGpE3Fo0Tgfvc1dErF3g+a/Jd/cE1gS+lx/vC9wXER/sMp/X5ZgrAtcCh0fEpSVjrU763ftP0u/LRsDxEfG1GuQ2hfS7djzwCOnv7GMRcU7JeF0dq6TR/h4F/DwiChVqSb8CziP9rN4G/A/wfdIxvz4i9igYbyGLT0WzFnA3EBGxQZF4i4mIWt+Aq0e6P9LjAjGfAeYBt7fchh4/VSLedS33lyadLZ0DTC2TI3BN/lfAX1n04azW9yr5M7wGWKZsPODWlvtXjvZzKPn/Mg14D+kD74+kD4JlKvxdurPk6wY72dZhrNWBQ4BBUqHYM//ODAC3l4i3O3AucD3wEeA5efsKwJ/7nNuLgS8D/wecAGyVtz8fuKNfx5r//n8LXDTC7YkSebX+fd3Ztu+aEvE+BPwK2KJlW+Gf/2i3pam/GOX+SI87NQ/YMSLubN8h6a4S8ZZ9NqGIBcBBkj5J+sUq3BzTEisk/SLy/3p+XOaYp0n6V1Jz3tSIeLqLeGdL+i5wNHCupP8k/SHuACz28+xUPnvbj3QGdzXpbGl7YH/gtQXifI2Rfy8ErFoyvRUlbRAR8/J7rE86qyvjUuAM4M0RcXfL9kFJ3ywR7y3AlyPid60bI+JxSQf2ObevAd8hndU/0ZLbX8o0e1Ldsd4MvDsibmvfUfLvv7VZ8vQx9nUkIo6TdBbw5ZzPkZSvdYuZDEV/LUnHk/5oh+6TH88sGfMrwHRGLlLHlog3KGnXiPjV0IaIOFrSX4ATS8ZbKSL+ERH/NrRR0obAoyXiXUI6S4J0se65EXGfpDUpOBNgRBwh6QDgB8CGpG8zBwE/IbXfFibpXFIz1hnAmyLi3rzrLEmDBcON9fyisYZ8ELhY0jzS7926wLtLxtpk6EO8XUQcUzRYROw/xr4LC4arOrfXjLHvjBLxqjrWoxi9GL+/SE7ZT1v+Xp/9MJO0EelbTmH5Q3dvSbsDF5C+zVRiMrTpj/ofDRARp5WIuU1EFLqoV5d4kjTaH2aZeFXm1kXMN0TEL9q2TY2IJyt8j+VIHyg/Kvn6qcBQp4JbyuYmaQZwGLA5sNzQ9ojYoWS8bUhn1C8kfeOcAjwWEavUILeNgc8Bm7XFK9UmXdWxStozSl5P6Ec8ScsDG0bEDZW8QVXtRJPpBlzleP2PNVbMKt6HVBTeQPoGcR9wdsk4KwAfB76dH28M/EvJWL8GDiQ1MbwGOAU4potjHCRdyLw6H++7gM/VJLc/ADsC15G+HR0FHN3vY63z31cv4rXfJkOXze0lvbPl8dmSfptvpc5ArB5yV7eXActL2lLSVvn2Wrr4OivpNZK+BfyZVMR2AtaPiL1KhjwVeArYNj++B/jvkrFWj4iTgacj4pJIzXdd/R5HxFxgSkQ8ExGnArvWJLflIzW7KCLuiIijgDd2Ea/KY22sydCm/ymGt7NtAhxAupD2MdLF0qI2GKufdUTsPto+x6s01i6k/8u1gC+1bH+U9H9bmKS7SddqTgQ+HBGPSro9Ih4vEy/bMCL2kbQvPHvhsOzgpafzv/dKeiPwF6CbvtePS1oWuEbSscC9lB9/U3VuTyqNvbhN0sGkD8vSHRuo7lg3lXTdCNtF6t/w4hH2TeZ4w0yGor9KRNzU8vi2iJgDIOlzJWPOB47rOrNmxqssVqTrMadJektE/LiKmMDZpP7R+wDPSPop3fd8eCq3q6Z+rumCetnrDf+tNDr1Q6T26VVIF4rLegepqePgHGdtUi+XOuR2COkb2weAT5O+NYx5jW4cVR3r7cCbushjssUbZjJcyL0tIjYeZd/ciNioRMyrI2LL7rNrXryKY+0XEd+T9CFGKMwR8aURXtZJXJG6ee5LatOfRmrm+UVE/KNEvJ1Ibfqbkdq9twMOiIiLy+Rn/VXnv69exGs3Gc70b5H0xog4r3WjpH8Byo7avL37tBobr8pYQ33du/nKv5hIZzIXARdJWobU7jsL+AZpkemi8S6QdBVpxLCAQyKiUFfXMcYPDL3HBwrGu36ceB03AfQgt5+NE69Q82SVx5r9b8HnT/Z4w0yGM/2NSKMD/whclTe/DHglqQdF4X6wkvYca38U7H7VpHhV5zaRJC0fLYOEOnj+phFxi0YZth8RV420fZRYlXY9lrTuOPHah/GPFavq3Ebtn5/jXVIwXmXHmuMdOk68Qt8w6x5vsfh1L/rwbB/pt5P6DwdwI2nSpX0i4n0l4i0kTUdwzdCmlt0RLQOiHK+3ueWYx5J6wzxBGn7+YuCDEfG9MV84cqyNgSOAB0kXh78NvIr0+3JgRHQ8QEvSSRFxkKSLRtgdUbL/eo69QpcXl1tjrQtsHBG/ydcelo6IMoP4epHb8sA6UdFcSlUca8vv8C9J12aGXZSPiE8tSfEW06u+oL24AVsBXyB1xbsIOLhknDcDZ5L6/X4C2KjLvBoTr+rccsyhuYb+FTiZ1AZ/bclYfyCNEP4wqbfI3qSBQTsBl5eItxSwXbfH2BJvW+Am8hwtwEuAb3QR79+BK4E/5ccbAxfWJLc3kZpgb8+PXwrM7vex5jw+TyqsJwOvJ58Al8yr1vEWi19VoF7dSNO9Hgnckv+g30+JyZpGiT00K95Pc+zXOF5fYt2Y//0OsGu+X7boX9Nyf+5o+wrGLDWx3yixLif1OmmdpOuGLuJdQxqd2hrv+prkNof0Ad51blUfa8vrX0nqqXQzsHsF/7+1jhcxCQZnkYr9DqT2++0jTZ/6TEWx/wn8nTTd60q0DBV3vAmNNVvSLaRrNRfm6QD+WTLWwpb7j4yxr4gLJb2li775w0RE+6Re3fw+PxkRTw09kLQ0XXRRrTi3pyPi7+1v0UW8So81/55tCWxBmrb4b13kVvt4QyZD7509ST0vLlKat/pM2tq4isojeWcBWwO/Ab4aBdp6mxyvB7ktBfyM1Gz394h4RtLjQKE5yFsMDWwRsGHLIBcBZechfzdwKLBA0j9ZNEim8Pw2wF2SXglE7ll0COksrqxLJH2MNKp5J+A/SD/PMqrO7UZJbwOm5GstHyB1yCirkmOV9G/AW0knKmcDb42I0gW17vEWi5+/QtSepBVJhWBf0pn/6cC5EfHrErEWkuYD+QPpTGHYDyGKd1FrTLyqc8sxq+z7X3VPDwFrxwjTcJehtOrYV8nttKR+/4dExAMl4y1FGoOwc453Pmlhm8J/2D3IbQXSRfXW3D4dEaW+xVV1rPl3+AYWLVTS/jtctEtpreMtFn+yFP1WkqaTLtDtExE7lnh91V3UGhOv6txyzC+S5nI/p0yxKkPSpRGx7fjPTP3EI6LSRb6tf3rQpbTW8RaLPxmLfq+oy+l3mxyvm1iSHiVdGH6G1G2zm+aTTt+z428Xkk4Dvh4RV3bxfh6ctShevwdnjfY+awOzIuILS3K8ydCm31NK63fuQmo22hn4PWmdVseboFgRUdnC1kXetsBzXwG8XdIdwGNQauKroese25GmczgrP96b1E2yqH/J/w6NUxlalGQ/il/crDq3L+Z/R1xbuES8Ko91mHyxdO+c2/NJq8AtsfGA+nfZ7NWNNF/4t0jrsf6YtBbtCo7Xl9xE+gP+RH68NrB1j///O56znDQX/GK3ku97GWlA0dDjZYDLujiOxbqTFjm2HudW2drCVR4rsDJp4rfzSdOKHAfc3UVetY63WPyqAk2mG6n70x9Js/atnLfd7ngTn1t+/YmkhbNvzo+n07boeg9+B8osWP8cYJ2hW8n3vRVYreXxdFoWmy8R7xpaBo+R+nWXHY9QdW43Axu0PF5/6P+4n8dKakK8hDRSe6iJe14XedU6Xvutqc07VU+/26R4vZi6+BURsZWkqwEi4iGledMLk/TriNi5g6e+o0DM3UlnW88n9ZVel1TQNi+R4ueBq/PUDgJeTVpRqqwDgVOUpkQGeBgoPBVGj3IbaW3hg7qIV9WxfpRFE/D9QGkR8m7UPd5wVX16TLYb6ZfwdcBJpLPXR0l9Y1dyvAnP7XLSPOlX5cczKDkKtuzrxol5LbD6UOx87Cd3EW9NUvfjPYA1K8pxGjCtgjiV5gZMJU3n8BJgas2OdQPSYj3XkwYD/hfwgiU13tDNvXeAPBBl6ILkLhFRePrdpsarIpakt5O+OWwFnAbsBXw8yvUEmkead2dEUWIWUEmDETEg6Vpgy4hYKOnaiHhJ0VhWT5JeRPod3idKrNExmeK56LdRy/S7kn4cEWVXIWpcvG5iSdqUtIi2SJNolRoJKukB0nxAI43ajig3C+hvSE1anyPNx/834OUR8coyOdrkUGQsx2SK19Q2/VHF8PnWyw7bb2S8srEkHQ+cGREndPP+2R1lCvs49iB9vf4gaYrvacDRFb+H1U+3c13VMp6L/tiq/hrUpHhFYs0BPi5pE1I/5DMsVff4AAAWI0lEQVSj/Hw+lUyK1ioiHgOQtArl57V5Vh7f8Fxa/v6ii2ke8nw567XFO70muc0kXcBtjfe7LuJVdqwdqPPfV+l4LvrWd7FogfTVSAtdHyNpnRhlbeRx7FdtdiDp3cCnSGf7C8mDsyjxzUjS+0lThd/Holk/g7RwTJnczgA2JHVnHJoRM0hzU/U7t2NI12puasutVNGv8libzEV/bFWfNTYpXplYGwGbsqhLZBmXSRrpDKibqR0+DLwoCq6LO4pDgE2i5CRmIxgANotqLs5Vndubc7wnK4pX5bF2os5/X6XjTYb59CdUW5/Y/3K83seSdKyk20jt5DcAAxHxpjI5RMTKEbHKCLeVSxZ8SEstVrJ8IGkUc/sc8924gdTNsgpV5zaPNKq3KpUcq6QTc1PdeDoay1H3eIvFd++d4STdGRHrON7ExcrNJz+u4kw6NxGNKiIeLBFzS+BU0niCZ89ao9iU1EOLXW8ObAKc1xar6OLZQ5OarUxaXu+KtngdT2rWg9yGJnCbSeqffyElf245XmXHmuN9hDRI7MiI+H6R107GeIvFd9Efrs5Fte7xisaStGlE3CJpq5H2R8RVJXJYSBowtmBo0/CQUaYd/grS+gHX07L6VhSbkvrIMXZHRBTqDaQKp9/tQW77jxOvUBt8lcfaEnMm8CVSF9wTGf7/WmYsR63jtWpkm/5oRYZUIAp/HW1SvIpzO5R0RnPcCPuCtFhOUceTRsz+L/AD4A8VtAEvExGHjv+00UXEpwAk7d0+6EzS3iXiXZJfe0xEDGtKyxdQOy6EPcjttPzaQyLiq23xDikRr7JjbYl5j6TzgM+QFnBvvXBduKjWPV6rRp7p57lFRhURr3O83sfqFUkCXksawbg1aQWoEyPi9pLxPgv8mdRds7VZoUxT0VURsdV427qMd12UmGN+gnIrvUpaVccqaXPS2fNfgA9GxL1l8pks8do18kx/rMIk6RWONzGx2l5bWf/rfGZ/kdIEbrOATwO3Ad8umd6++d+Ptr4NxQag7Qa8AZiZB6MNWYVFTVEdk/Re0hqxG2jROsCQ2r0LrUPbg9z2Bd4GrC9pdltuZT4oKzvW7GzSMpCFl1qdpPGGaeSZ/ljq3GZe93hdXMgdsf910Qt+OdbQWsr7kCZuOwf4YTcDjKog6SWki5BHA59s2fUocFFEPFQw3jTS1MefAw5vjVf0G0gPclsXWH+k3IDrIqLQB0mVx5rjTR2pG6mk7YF9I+J9I7xs0sZbLI6L/nCS7oqItR1v4mJJupmK+l9Leox0Vn9m/ndYzG4vgrW8z5oR8dcSr1smIp6uIoeWmJWMou1FblWr6lhb4m1J+layN2nBknMi4mtLajxoaPPOOGoxVHqSxisba6j/dRVtlz/KeWySb626vgjW4mTgjSVet7Wko1g0NcHQoLFS8x5JOpg0530Vo2irzm1P4BjS4jNqiVdqvERVxyrpBaQmu32B+0nLQ6rs9ai6x1ssfhPP9DX6ws0CdoiIFR1vYnLLMS+igv7Xk4GkW0gTt81hUVMWUXIUrKS5pEVouh5F26Pc3hQlZ0wdJV7Xx5q79f4eODAi5uZt87r4cKt1vHZNPdP/Ysl9jld9btDd6kzDaNFAoxFFgYFGknaIiN/m++u39v6RtGfJpqK/R8QvS7xuNFWOoq06t/uqKvhZVce6J+kC/0WSfkVqCuxmioS6xxumkWf6tuRShQONWrsItncXLNuVUdLnSauEncPwbzWFB6LleCdTwSjaHuX2VVKz3U/a4pVqYqvyWHO8oYv++5LGhJwOnFu210zd4z0bt4lFX9L1jNH+XKLfb2PiVZ1bjvloS8xlSYO8Hivb9jvG+/xnRHylwPOf7VPe3r+8bH/zUcY5RESUGYg26odc5AFXBWNVndupo8Qrtd5Blcc6QuzppIul+0TEjktyvKYW/XXH2h8Rdzhe72ONEl+ks5ttIuLw8Z5fMHbRaSIqP9PvFUkrAUTEP/qdS691e6ySlgPeQ5rV9XrSeseFxyNMlniLxW9i0R+JpDWAB6roNti0eFXnlmOWHrk5RsxCXUolPUya+13Aq1g0D7yA7SNieokcppHmrH913nQJcHRElGqrVlo79QxgaKK5+4F3RsSNNchtLeBrwHZ50+9Jg47uLhmvkmNVmg326ZzPbqTV1gpPDzFZ4i0mKlidfrLdgG2Ai0ltl1uSugz+lbT26a6ON3G55Zh7ttz2Aj4PXNqD//c7Cz7/NWPdSubwY9KCLBvk25Gkvtdlj+mPwOtaHr8W+GNNcrsAeBepw8jSwAHABf0+VuD6lvtLA1d1+XtV63iLxa8y2GS5AYPAzqQ2sodITQmQFvC42vEmLrf82lNbbt8GjgCeUzLWo8AjI9weBRbU4Hfvmk62FYh3bSfb+pRbLY+1vYhWUKRrHa/91tQum0tHvgIu6eiIuAwg0jS/jjexuRER7yr1wpFjrVxVLEl7AGtFXrBd0uWkqR0ADouIs0uEfULS9hHxhxxzO+CJcV4zlnmSPkFq9oC0XOS8krGqzu0BSfuRZjuF1Aulmz72VR3rSyQ9ku8LWD4/Ljt4rO7xhmlq0V/Ycr/9l7pMu3ST4lWd24gk/UtE/LyqeCUdRuovPWQq8HJgRdK3kjJF/72k9YCnkf6IHyQ1e5T1b6QmmaFukL/P28roRW5fA75M+t34I6m5p5t4XR9rREzpIodJF69dIy/kSnoGeIz8KcqipfAELBcRReeYb0y8qnMb430+FRFj9bnvOUlXRsTLWx5/PSIOzvcvi4htuoi9CkBEPDLecydanXOz7jWy6Jt1QtLciNholH1/iogNS8RcFXgni08jXXhG0RxvAPjYCPHKjJeoOrf1gfePEK/U9BpVHmuTNbV5x2pA0qvH2h8Rvxtr/wS4XNK/R8SwefiV1vS9omTMXwCX0bb0Yhf+B/hIRfGqzu0npInpflZRvCqPtbF8pm99ozR5W7uhWRPX7nXb5ngkPYdFUwgMTUXwMlLb/psj4r4SMSsd1CXpDxGxfUWxqs7t8ogovbDOCPEqO9Ymc9G32si9RT5OWjDjMxEx0ofChJO0A7B5fnhj5EnYSsb6IPAP4Od0ufRijrcjqVfMhXQ5v00PcnsbsDFpucoq5vKp7FibzM071nf5j/kTpLP8z0bEBX1OaZhc5Idm21wxd0PcNyLKzKf/FPAF0liEoTOuoMDSi23eRRojsQzdL55ddW5bAO8gTRbWmlupuXyo9lgby2f61jeS3kgqMH8nndn/oc8pjUjSsqQFU94G7EIauXpOmW8ikuYBW0fE/RXldmtEtC8WUzZW1bnNJa2I9lRF8So71ibzmb7108+Au0kDdg6TdFjrzrK9PKoiaWdSc8LOwEWkqW1f3uVgsrks6uZahT9K2iwibqogVtW53QCsSpqiowpVHmtjuehbP1Wy/FsP/Yo0AGj7yAuoKM0R343HgGvyNMat7dKlukWS5kK6RtLtOd7QqM0y3Rirzm1V4BZJV1LNimhVHmtjuehb30TEJfDsVLJD/eHnRsQ/+5fVMFuRRuT+Jjd9nElaZKQbP8m3quxaYayqc6t6cF2Vx9pYbtO3vpG0NPBZ0lD6O0hnbmuTpjg4IiKe7mN6w0h6Jamp5y3AtaQVjE7qb1ZmxbnoW99I+jKwMvDBiHg0b1uFtNbuE1HlHOIVkbQU8HpgVpRcAcqsn1z0rW8k3Qa8INp+CSVNAW6JiI37k9mzeYw5UKlsf3OzfnKbvvVTtBf8vPEZSXU4GzlujH3d9Dc365ul+p2ANdpNkt7ZvjEPfrqlD/kMExGvG+NWqOBLWlPSiZJOkLS6pKMkXS/ph5KeVzQ3SWtLOlPS7yV9TNIyLfsKXYztQW6rSPqcpDPyqNzWfd8oEa+yYzUXfeuv9wHvk3SxpOPy7RLgA6S53WtJ0k6Sio4a/i5wE3AXqc//E8AbSF1Cv1kijVNIy1a+H3gecImk1fO+MRevn4DcTiVdlP8xMEvSjyVNzfvKTEdd5bFap0ts+eZbr26kZpL359uO/c6nLa//I81H8z3StAKDwBxgz4Kxrm65f2fbvsJLCLa/hrSK1I3AhhRcXm8CcjsC+F9g9aK5VX2svjV3uUSrAUmr5bvX5Nuw7VFyoq8KHQccBFwK7Jb/PTwivl4iVuu36tPH2NepZSQtF3lMQ0R8T9JfgfNJK3v1M7epkpaKiIU5t89Iugf4HbBSiXhVHmvjuXnH+mkOi86cB1tuQ4/7LSLi4oh4MiJ+AtxTsuAD/FTSSjnox4c2StqI9G2iqO8Aw6YtjojfkBasv6HPuf2MtovcEfFd4EOkSd2KqvJYG89dNq1vJK0bEXf0O4/R5FG4H27Z9AXSIh6Ap/S1yclF3/qm6kU7qibp1DF2R1Q0OEvSVlFhn39VuKh8D3Kr7bE2hdv0rZ/U7wTGEt3NplnEe4F/rzDey0kLoVSh6tzqfKyN4DN96xtJfyNNYjaiKD+7Y2Xy6ODpkeeYz3PrH0CaOuKF/czNrAyf6Vs/PUG6aFtLkmYB3wIey1NGfIbUZ/xK4O0l4k0jzRQ5M2+6Bzg/Ih4umd8qwIyI+FPb9hdHxHUFY60JEBF/lTQDeBVwa0TcWCa3EeJ/NiI+VlGs9YEtgZsiou+D+CYbn+lb30yCNv0bSAugz83z8FwK7BXlVsx6J2mq4V+Tij3AWsBOwKcior2r5Hjx3gp8hbRAyTLAARFxZd5X6Ocq6d3A4aTmtmNI32RuALYHjo2Ikwvmdnz7JtKyiadD8W9wkn4SEW/O9/cgHffFwCuBz+WeQdYhn+lbP1WyjF4PPRURcyFNribptjIFPzsCeFn7Wb2k6cDlLN4/fjwfy/HulbQ1cIakj0bEuRS/VnIwaeH35UlTXG+Uz/ink0boFir6wL8Cl5A+4IZymUX5b3Wto27/C9ghIm6XtAZpkfTvlozbSC761jcRsdiQfEkbktainRURm098VsM8R9KhLY9XbX0cEV8qEEssWmy81ULKXdCeEhH35jyukPQ64OeS1h7lfcbydEQ8Djwu6U8R8dcc96GSE99tBnya1JT14Yj4i6QjI+K0ErFg+PEsHXkVs4i4X9LCUV5jo3DRt76T9HxgH1Kx3wL4HOnMsN++TZrvf6THRYvhZ4CrJP2aNMcNwDqk5p1Pl8jtUUkbDrXn5zP+15JWvir6YRmSlom0aM0bhzbmFc0KD+CMtDbCf0p6GfA/ks4rE6fFSyQ9QvpwnCrpefl4l6X7lcwax2361jeSDiKtRjUT+GG+/TQi1u9rYh2Q9PKhNvQCr5kO7MLiF3IfKvH+LwEeG2p+atm+DPDWiPifArHWAf4SEQvats8EXphHv5YiScB/ANtGxH5l44wSe1VSfpdWGXdJ56JvfSPpKdLF0Q9FxGDeNi8iNuhvZiOTtBnpQ2pf4OGIGOjBe1waEdvWMV6dc+tFvCWVm3esn55Hmj/luNxl8Ieknii1IWk9FhX6p0kXFQci4s89esvlahyvzrn1It4SyROuWd9ExAMR8c2IeA2wI/AwcJ+kmyV9ts/pIelS4DzSydFbIuJlwKM9LPhQ/FrBRMarc269iLdEctG3WoiIuyPiuNxksjvwz37nBNxHunD7XGBG3ubCYpOai771jaT9JL1jhF3bAHNH2D6h8oCgLUj9y4+SdDswPfeL75Wq5yOqMl6dc+tFvCWSi7710/uBc0fYfg5p7vW+i4i/R8SpEbEz6cPok8CXJd01zkuHyWvQrtLBU0f6EOxpvDrn1ot4Teeib/20TET8o31jRDxGzS7oAkTEfRHxtYjYjjRFQRHzgDntC4WP8B6dLgpSZbw659aLeI3mLpvWN5JuJvWEeaxt+8rAlRGxaX8yezaP2WPtj4jdC8abCXwJWAM4kTQadyhW4QVZqoxX59x6Ea/J3GXT+ulk4GxJ7xlaQSt3kTyB4vO99MK2pNGzPyDNj9NVm3FE3JNHp34GeBOLCleQmrT6Fq/OufUiXpO56FvfRMQXJf0D+F1eo1XAo8DnI+LE/mYHwJqkaRL2JU0RcR7wgzLTDUvanHSG+hdg66F5c8qqMl6dc+tFvKZz847VQm7SGZq3pXYkTSUV/y+QpkIutEB6bso6JCJ+XVE+lcWrc269iNd0LvrWN20zWEL6qn4/8IehmRT7LRf7N5IK/nrAbOCUiLhnrNeNFCcinhxh+/bAvhHxvn7Fq3NuvYjXdG7esX5aeYRt6wFHSDoqIkZdSnEiSDodeBHwC9LZfeneIa1FS9KWpOaivYHbKdfGXVm8OufWi3hN5zN9qx1JqwG/KbL6U4/yWAgM9Sxq/UMREBHRSd/xoVgvYNEcPvcDZ5Hmml93zBdOQLw659aLeE3nom+1JOnqiNiy33lUJX+A/B44cGg65G5mFK0yXp1z60W8pvPgLKudvApU4Tnma25P4F7gIknflrQj3XUBrTJenXPrRbxG85m+9Y2k61l8ArPVSF3z3hkRt0x8Vr0laUVgD1JTxQ6ktXHPLdszpcp4dc6tF/GaykXf+kZSe5tsAA+0j9BdUuWVtPYG9omIHesUr8659SJek7joW99J2gIYmnLh5iVxDhWl9WbfA2wEXA+c3L48Yb/i1Tm3XsRrOhd96xtJ04CfAmsD15HaabcA7gT2iIhH+phepSSdRVp56/fAbsAdEXFIHeLVObdexGs6F33rG0nHA08Bh0XEwrxtKeDzwPIR8f5+5lclSddHxBb5/tLAFd10Sa0yXp1z60W8pvPgLOun1wMvHir4ABGxUNLHSF/jlyRPD92JiAVS151PqoxX59x6Ea/RfKZvfSPpmoh4adF9k5GkZ1g00EvA8sDjlBjoVXW8OufWi3hN5zN966fl8rD69lM3AVP7kE/PRMSUusarc269iNd0PtO3vpF0MWMsNB4Rr5u4bMyawUXfzKxBPA2D9Y2kw1ru792277MTn5HZks9F3/ppVsv9j7bt23UiEzFrChd96yeNcn+kx2ZWARd966cY5f5Ij82sAr6Qa33T0v+6te81+fFyEbFMv3IzW1K56JuZNYibd8zMGsRF38ysQVz0rZEkhaTjWh5/WNJRfUzJbEK46FtTPQnsKWmNfidiNpFc9K2pFgAnAR9s3yHpTZIul3S1pN9Iem7efpSk0yT9XtIdkvaUdKyk6yX9StIy+Xkvk3SJpDmSzpf0vIk9NLPRuehbk50AvD2v4NXqD8A2EbElcCZwWMu+DUmLcu8OfA+4KC/w8QTwxlz4vwbsFREvA04BPtPbwzDrnKdWtsaKiEcknQ58gFS0h6wFnJXP0JcFbm/Z98uIeFrS9cAU4Fd5+/XAesAmwIuAC/JiH1OAe3t5HGZF+Ezfmu4rwIHAii3bvgZ8PZ/BvxtYrmXfk5BW+AKejkUDXRaSTqIE3BgRL823LSJi514fhFmnXPSt0SLiQeCHpMI/ZBpwT76/f8GQtwIzJG0LIGkZSZt3nahZRVz0zeA4oLUXz1HAjyTNAe4vEigingL2Ao6RdC1wDfDKivI065qnYTAzaxCf6ZuZNYiLvplZg7jom5k1iIu+mVmDuOibmTWIi76ZWYO46JuZNcj/B4KzglXGT4QlAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x7fe5aeeacf28>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "g_pred_ = model.predict(row)['p1'][0, 0]\n",
    "plot_local_contrib(row, local_glm2, X, g_pred=g_pred_, scale=True)                                                             "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Shutdown H2O\n",
    "After using h2o, it's typically best to shut it down. However, before doing so, users should ensure that they have saved any h2o data structures, such as models and H2OFrames, or scoring artifacts, such as POJOs and MOJOs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Are you sure you want to shutdown the H2O instance running at http://127.0.0.1:54321 (Y/N)? y\n",
      "H2O session _sid_898f closed.\n"
     ]
    }
   ],
   "source": [
    "# be careful, this can erase your work!\n",
    "h2o.cluster().shutdown(prompt=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Summary\n",
    "In this notebook, LIME was used to explain and generate reason codes for a complex GBM classifier. To do so, local linear models were fit to appropriate, representative samples and linear model coefficients were used to explain the average behavior in the samples and to create reason codes. Reason codes were assesed against domain knowledge and reasonable expectations. A ranked prediction plot was also introduced to compare surrogate linear model predictions to GBM model predictions. These techniques should generalize well for many types of business and research problems, enabling you to train a complex machine learning model and analyze, validate, and explain it to your colleagues, bosses, and potentially, external regulators."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}