examples/model_explainability/interpret_dnn_layers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Copyright (c) Microsoft Corporation. All rights reserved.*\n",
    "\n",
    "*Licensed under the MIT License.*\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Understand your NLP models\n",
    "\n",
    "\n",
    "0. [Methodology](#0-Methodology)\n",
    "\n",
    "    - 0.1 [Multi-level Quantification](#0.1-Multi-level-Quantification)\n",
    "    - 0.2 [Perturbation-based Approximation](#0.2-Perturbation-based-Approximation)\n",
    "    \n",
    "    \n",
    "1. [How to understand a simple model](#1-How-to-understand-a-simple-model)\n",
    "\n",
    "    - 1.1 [Prepare necessary components](#1.1-Prepare-necessary-components)\n",
    "    - 1.2 [Create an Interpreter instance](#1.2-Create-an-Interpreter-instance)\n",
    "    - 1.3 [Train the Interpreter](#1.3-Train-the-Interpreter)\n",
    "    - 1.4 [Show and visualize the results](#1.4-Show-and-visualize-the-results)\n",
    "    \n",
    "    \n",
    "2. [How to understand a saved PyTorch model](#2-How-to-understand-a-saved-PyTorch-model)\n",
    "\n",
    "    - [2.1 Prepare necessary components](#2.1-Prepare-necessary-components)\n",
    "    - [2.2 Create an Interpreter instance](#2.2-Create-an-Interpreter-instance)\n",
    "    - [2.3 Train the Interpreter](#2.3-Train-the-Interpreter)\n",
    "    - [2.4 Show and visualize the results](#2.4-Show-and-visualize-the-results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "from tempfile import TemporaryDirectory\n",
    "\n",
    "sys.path.append(\"../../\")\n",
    "import json\n",
    "import torch\n",
    "import logging\n",
    "from torch import nn\n",
    "from urllib import request\n",
    "import scrapbook as sb\n",
    "from pytorch_pretrained_bert import BertModel, BertTokenizer\n",
    "\n",
    "# import utils\n",
    "from utils_nlp.interpreter.Interpreter import calculate_regularization, Interpreter\n",
    "\n",
    "# disable the inner message of pytorch_pretrained_bert\n",
    "logging.getLogger().setLevel(logging.WARNING)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a tutorial on how to utilize the `Interpreter` class to explain certain hidden layers in your NLP models. We provide the explanation by measuring the information of input words ${\\\\bf x}_1$,...,${\\\\bf x}_n$ that is encoded in hidden state ${\\bf s} = \\Phi({\\bf x})$. \n",
    "\n",
    "The method is described in our *ICML 2019* paper: [Towards a Deep and Unified Understanding of Deep Neural Models in NLP](https://www.microsoft.com/en-us/research/publication/towards-a-deep-and-unified-understanding-of-deep-neural-models-in-nlp/).\n",
    "In this torturial, we provide two examples for you to get started quickly.\n",
    "\n",
    "## 0 Methodology\n",
    "\n",
    "We briefly introduce our algorithms here. In short, we are trying to use Mutual Information to understand $\\Phi$, the model or layer we want to understand. You can also refer to our paper [here](https://www.microsoft.com/en-us/research/publication/towards-a-deep-and-unified-understanding-of-deep-neural-models-in-nlp/) for more details on algorithm.\n",
    "\n",
    "### 0.1 Multi-level Quantification\n",
    "\n",
    "Suppose the input random variable is $\\bf X$ and the hidden random variable ${\\bf S} = \\Phi({\\bf X})$. We can provide a global/corpus-level explanation by evaluating the mutual information of $\\bf X$ and $\\bf S$:\n",
    "\n",
    "$$MI({\\bf X};{\\bf S})=H({\\bf S}) - H({\\bf H}|{\\bf S})$$\n",
    "\n",
    "Where $MI(\\cdot;\\cdot)$ is the mututal information. $H(\\cdot)$ stands for entropy. Because $H({\\bf S})$ is a constant only related to input dataset $\\bf S$, the only thing we need to consider is $H({\\bf H}|{\\bf S})$. This conditional entropy can be seen as the global/corpus-level information loss when r.v. $\\bf X$ is processed by $\\Phi$. By definition:\n",
    "\n",
    "$$H({\\bf X}|{\\bf S}) = \\int_{{\\bf s}\\in {\\bf S}}p({\\bf S})H({\\bf X}|{\\bf s})d{\\bf s}$$\n",
    "\n",
    "Then, we can decompose the corpus-level information loss to sentence-level:\n",
    "\n",
    "$$H({\\bf X}|{\\bf s}) = \\int_{{\\bf x'}\\in {\\bf X}}p({\\bf x}'|{\\bf s})H({\\bf x}'|{\\bf s})d{\\bf x}'$$\n",
    "\n",
    "If we make an assumption that the inputs of $\\Phi$ are independent, we can further decompose the sentence-level information loss to word level:\n",
    "\n",
    "$$H({\\bf X}|{\\bf s}) = \\sum_i H({\\bf X}_i|{\\bf s})$$\n",
    "$$H({\\bf X}_i|{\\bf s}) = \\int_{{\\bf x'}_i\\in {\\bf X}_i}p({\\bf x}_i'|{\\bf s})H({\\bf x}_i'|{\\bf s})d{\\bf x}_i'$$\n",
    "\n",
    "Note that $H({\\bf X}_i|{\\bf s})$ stands for the information loss when word ${\\bf x}_i$ reaches hidden state $s$. Therefore, we can use this value as our explanation. Higher value stands for the information of corresponding word is largely lost, which means that this word is less important to $\\bf s$, and vice versa.\n",
    "\n",
    "### 0.2 Perturbation-based Approximation\n",
    "\n",
    "In order to calculate $H({\\bf X}_i|{\\bf s})$, we propose a perturbation-besed method. Let $\\tilde{\\bf x}_{i}={\\bf x}_{i} +{\\boldsymbol \\epsilon}_{i}$ denote an input with a certain noise $\\boldsymbol{\\epsilon}_{i}$. We assume that the noise term is a random variable that follows a Gaussian distribution, ${\\boldsymbol{\\epsilon}_{i}}\\in \\mathbb{R}^{K}$ and ${\\boldsymbol \\epsilon}_i\\sim{\\mathcal N}({\\bf0},{\\boldsymbol\\Sigma}_{i}=\\sigma_{i}^2{\\bf I})$. \n",
    "In order to approximate $H({\\bf X}_i|{\\bf s})$, we first learn an optimal distribution of ${\\boldsymbol{\\epsilon}} = [{\\boldsymbol{\\epsilon}}_1^T, {\\boldsymbol \\epsilon}_2^T, ..., {\\boldsymbol \\epsilon}_n^T]^T$ with respect to the hidden state \n",
    "${\\bf s}$ with the following loss:\n",
    "\n",
    "$$L({\\boldsymbol \\sigma})=\\mathbb{E}_{{\\boldsymbol \\epsilon}}\\Vert\\Phi(\\tilde{\\bf x})-{\\bf s}\\Vert^2-\\lambda\\sum_{i=1}^n H(\\tilde{\\bf X}_{i}|{\\bf s})|_{{\\boldsymbol\\epsilon}_{i}\\sim{\\mathcal N}({\\bf 0},\\sigma_{i}^2{\\bf I})}$$\n",
    "\n",
    "where $\\lambda>0$ is a hyper-parameter, ${\\boldsymbol \\sigma}=[\\sigma_1,...,\\sigma_n]$, and $\\tilde{\\bf x} = {\\bf x} + \\boldsymbol{\\epsilon}$. The first term  on the left corresponds to the maximum likelihood estimation (MLE) of the distribution of $\\tilde{\\bf x}_{i}$ that maximizes $\\sum_{i}\\sum_{\\tilde{\\bf x}_{i}}\\log p(\\tilde{\\bf x}_{i}|{\\bf s})$, if we consider $\\sum_{i}\\log p(\\tilde{\\bf x}_{i}|{\\bf s})\\propto -\\Vert\\Phi(\\tilde{\\bf x})-{\\bf s}\\Vert^2$. In other words, the first term learns a distribution that generates all potential inputs corresponding to the hidden state ${\\bf s}$. The second term on the right encourages a high conditional entropy $H(\\tilde{\\bf X}_{i}|{\\bf s})$, which corresponds to the maximum entropy principle. In other words, the noise $\\boldsymbol \\epsilon$ needs to enumerate all perturbation directions to reach the representation limit of ${\\bf s}$. By minimizing the loss above, we can get the optimal ${\\sigma}_i$, then we can get the $H(\\tilde{\\bf X}_i|{\\bf s})$:\n",
    "\n",
    "$$H(\\tilde{\\bf X}_{i}|{\\bf s})=\\frac{K}{2}\\log(2\\pi e)+K\\log\\sigma_{i}$$\n",
    "\n",
    "Then, we can use $H(\\tilde{\\bf X}_i|{\\bf s})$ to approximate $H({\\bf X}_i|{\\bf s})$. Again, you can refer to our paper [here](https://www.microsoft.com/en-us/research/publication/towards-a-deep-and-unified-understanding-of-deep-neural-models-in-nlp/) for more details on algorithm."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1 How to understand a simple model\n",
    "\n",
    "In this section, we use a simple linear function as an example to help you be familiar with the usage of Interpreter utils."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1 Prepare necessary components\n",
    "Suppose the $\\Phi$ we need to explain is a simple linear function:\n",
    "$$\\Phi(x)=10 \\times x[0] + 20 \\times x[1] + 5 \\times x[2] - 20 \\times x[3] - 10 \\times x[4]$$\n",
    "From the definition of $\\Phi$ we can know that, the weights of the 2nd and the 4th elements in input $x$ are the biggest (in abs form), which means that they contributes the most to the results. Therefore, a reasonable explanation should show a similar pattern."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "device = torch.device(\"cpu\" if not torch.cuda.is_available() else \"cuda\")\n",
    "\n",
    "# Suppose our input is x, and the sentence is simply \"1 2 3 4 5\"\n",
    "x_simple = torch.randn(5, 256) / 100\n",
    "x_simple = x_simple.to(device)\n",
    "words = [\"1\", \"2\", \"3\", \"4\", \"5\"]\n",
    "\n",
    "# Suppose our hidden state s = Phi(x), where\n",
    "# Phi = 10 * word[0] + 20 * word[1] + 5 * word[2] - 20 * word[3] - 10 * word[4]\n",
    "def Phi_simple(x):\n",
    "    W = torch.tensor([10.0, 20.0, 5.0, -20.0, -10.0]).to(device)\n",
    "    return W @ x\n",
    "\n",
    "\n",
    "# Suppose this is our dataset used for training our models\n",
    "dataset = [torch.randn(5, 256) / 100 for _ in range(100)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2 Create an Interpreter instance\n",
    "\n",
    "In the following, we'll show you how to calculate the $\\sigma_i$ using functions in this library. To explain a certain $\\bf x$ and certain $\\Phi$, we need to create an Interpreter instance, and pass your $\\bf x$, $\\Phi$ and regularization term (which is the standard variance of the hidden state r.v. $\\bf S$) to it. We also provide a simple function to calculate the regularization term that is needed in this method."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Interpreter()"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# calculate the regularization term\n",
    "regularization_simple = calculate_regularization(dataset, Phi_simple, device=device)\n",
    "\n",
    "# create the interpreter instance\n",
    "# we recommend you to set hyper-parameter *scale* to 10 * Std[word_embedding_weight]\n",
    "# 10 * 0.1 in this example\n",
    "interpreter_simple = Interpreter(\n",
    "    x=x_simple,\n",
    "    Phi=Phi_simple,\n",
    "    regularization=regularization_simple,\n",
    "    scale=10 * 0.1,\n",
    "    words=words,\n",
    ")\n",
    "interpreter_simple.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3 Train the Interpreter\n",
    "\n",
    "Then, we need to train our interpreter (by minimizing the loss [here](#0.2-Perturbation-based-Approximation)) to let it find the information loss in each input word ${\\bf x}_i$ when they reach hidden state $\\bf s$. You can control the iteration and learning rate when training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 5000/5000 [00:05<00:00, 976.01it/s] \n"
     ]
    }
   ],
   "source": [
    "# Train the interpreter by optimizing the loss\n",
    "interpreter_simple.optimize(iteration=5000, lr=0.5, show_progress=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.4 Show and visualize the results\n",
    "\n",
    "After training, we can show the sigma (directly speaking, it is the range that every word can change without changing $\\bf s$ too much) we have got. Sigma somewhat stands for the information loss of word ${\\bf x}_i$ when it reaches $\\bf s$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.00316059, 0.00158621, 0.00629779, 0.00158636, 0.0030826 ],\n",
       "      dtype=float32)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Show the sigma we get\n",
    "sigma_numbers = interpreter_simple.get_sigma()\n",
    "sigma_numbers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAagAAACsCAYAAAAuVDhiAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAALtUlEQVR4nO3dXYwddRnH8d9v20JJKRZtNYUSkcSQADG0KVVTQwwSLdCgl1xojNEQIxclXBAJiRETr0wMXmka0EhAQXnxAg2CESKIUvvKi6UGsMQGkqVBXiqJCDxenNntnNnZ0132zM4zs99PctLZ//znP888e3p+e84sxREhAACymWi7AAAA6hBQAICUCCgAQEoEFAAgJQIKAJDS8vlMPmn1mli57oymalkSjr3yetsl9MLGcz/cdgmdt+/QZNsldN6qtae1XUIvHDv87NGIWFcdn1dArVx3hi76/m3jq2oJenzn79ouoRf+/IcdbZfQeadf+qO2S+i8jV/b1nYJvfDoVz/5Yt04H/EBAFIioAAAKRFQAICUCCgAQEoEFAAgJQIKAJASAQUASImAAgCkREABAFIioAAAKRFQAICUCCgAQEoEFAAgJQIKAJASAQUASImAAgCkREABAFIioAAAKRFQAICUCCgAQEoEFAAgJQIKAJASAQUASImAAgCkREABAFIioAAAKRFQAICUCCgAQEoEFAAgJQIKAJASAQUASImAAgCkREABAFIioAAAKRFQAICUCCgAQEoEFAAgJQIKAJASAQUASOmEAWX7atu7be9++81/L0ZNAACcOKAiYmdEbI6IzSetPn0xagIAgI/4AAA5EVAAgJQIKABASgQUACAlAgoAkBIBBQBIiYACAKREQAEAUiKgAAApEVAAgJQIKABASgQUACAlAgoAkBIBBQBIiYACAKREQAEAUiKgAAApEVAAgJQIKABASgQUACAlAgoAkBIBBQBIiYACAKREQAEAUiKgAAApEVAAgJQIKABASgQUACAlAgoAkBIBBQBIiYACAKREQAEAUiKgAAApEVAAgJQIKABASgQUACAlAgoAkBIBBQBIiYACAKTkiJj7ZPsVSS82V86CrZV0tO0ieoA+Lhw9HA/6uHBd6OFHI2JddXBeAZWd7d0RsbntOrqOPi4cPRwP+rhwXe4hH/EBAFIioAAAKfUtoHa2XUBP0MeFo4fjQR8XrrM97NU9KABAf/TtHRQAoCcIKABASr0IKNs/tT1p++m2a+kq22fZftj2QdvP2N7Rdk1dZHul7V22DxR9vKntmrrK9jLb+2zf33YtXWX7sO2nbO+3vbvteuarF/egbF8s6Zik2yLigrbr6SLb6yWtj4i9tldL2iPpSxHx95ZL6xTblrQqIo7ZXiHpMUk7IuKvLZfWObavk7RZ0mkRsb3terrI9mFJmyMi+3+oW6sX76Ai4k+SXm27ji6LiJcjYm+x/aakg5LObLeq7omBY8WXK4pH938KXGS2N0i6QtItbdeC9vQioDBets+WtFHSE+1W0k3FR1P7JU1Keigi6OP83SzpeknvtV1Ix4WkB23vsX1128XMFwGFIbZPlXSPpGsj4o226+miiHg3Ii6UtEHSFtt87DwPtrdLmoyIPW3X0gNbI2KTpMskXVPcDukMAgrTinsm90i6IyLubbuerouI1yQ9Imlby6V0zVZJVxb3T+6UdInt29stqZsi4qXiz0lJ90na0m5F80NAQdL0zf1bJR2MiB+2XU9X2V5ne02xfYqkSyU9225V3RIRN0TEhog4W9JVkv4YEV9uuazOsb2q+IUn2V4l6fOSOvWbzr0IKNu/lPQXSefaPmL7623X1EFbJX1Fg59W9xePy9suqoPWS3rY9pOS/qbBPSh+TRpt+Iikx2wfkLRL0m8j4oGWa5qXXvyaOQCgf3rxDgoA0D8EFAAgJQIKAJASAQUASImAAgCkREABAFIioAAAKRFQAICUCCgAQEoEFAAgJQIKAJASAQUASImAAgCkREABAFIioAAAKRFQAICUCCgAQEoEFAAgJQIKAJASAQUASImAAgCkREABAFJaPp/Jyz748Yh33qrZY8l1R3jojxlf1I3PWGeua9QU4FHjg43Zlh1VytTGnC6rppZR56xMHRqd03E1C4xqkSR5xATPcszUjjl9u2oWm0/vRn+LY7ZTzL5O8UX12FFzZ6ttuoYTPXdGjNv1dZxwjaG+zfFaVN+j40Mz15n5PZ7lez/b/Bn7Z/+Gjzp2lmdPuaQT1DK6svq9o5+c81l/eNZcn6gz/uYvfP3q3BO9QNTuPWG3Ru+t/J3au2f/7yNiW/WIeQVU/O8tnXzRN6WJ4o3X1LPCllyMTbg0Xpo3URor759wZa3S/gkPrzG9Vmmseq7q+UesZbs01aVSPD3NdXOnxiaG506MmuvBOWauq9IxLpVat3/m3HL9xy/Vs5y35vwTM2udqPSlttbaayn1cKK+F/XXVbru4sX6+LXE9JN4wjH0NBi9P6Y/Hphas7z+RCmghtaqWbd2vyrnKm0P9tdvD9VaV/ds+0vXMuO65nKuurrncK7jax5/2Rvanno5nF6/9P2ubE/Nm9qurqvi69ptlZ5bNWtVt6ePKa1Vfemujk2fbagmT69V/kHOlbXk8tj06PT1lUctV84xtbdcf+m8lVfz6lj1vB66Bg/VWB0b1FKzbqn+2dadGiv3bcbVVsbqrkWSTlm+Zq1q8BEfACAlAgoAkBIBBQBIiYACAKREQAEAUiKgAAApEVAAgJQIKABASgQUACAlAgoAkBIBBQBIiYACAKREQAEAUiKgAAApEVAAgJQIKABASgQUACAlAgoAkBIBBQBIyREx98n2A5Jq/9/xLVgr6WjbRfQUvW0GfW0OvW3GYvX1aERsqw7OK6Aysb07Ija3XUcf0dtm0Nfm0NtmtN1XPuIDAKREQAEAUupyQO1su4Aeo7fNoK/NobfNaLWvnb0HBQDoty6/gwIA9BgBBQBIKU1A2d5m+5Dt52x/u2b/ybbvKvY/Yfvs0r4bivFDtr9QjJ1l+2HbB20/Y3vH4l1NHg30daXtXbYPFH29afGuJpdx97a0b5ntfbbvb/4q8mmir7YP237K9n7buxfnSnJpqK9rbN9t+9nitfbTYy06Ilp/SFom6XlJ50g6SdIBSedV5nxL0k+K7ask3VVsn1fMP1nSx4p1lklaL2lTMWe1pH9U1+z7o6G+WtKpxZwVkp6Q9Km2r7UPvS0dd52kX0i6v+3r7EtfJR2WtLbt6+thX38u6RvF9kmS1oyz7izvoLZIei4iXoiItyXdKemLlTlf1KAZknS3pM/ZdjF+Z0T8NyL+Kek5SVsi4uWI2CtJEfGmpIOSzlyEa8mkib5GRBwr5q8oHkvxN23G3ltJsr1B0hWSblmEa8iokb5i/H21fZqkiyXdKkkR8XZEvDbOorME1JmS/lX6+ohmhsn0nIh4R9Lrkj40l2OLt6obNfhpfylppK/FR1D7JU1KeigillpfpeaeszdLul7Se+MvuROa6mtIetD2HttXN1B3dk309RxJr0j6WfGR9C22V42z6CwB5Zqx6k/ls80ZeaztUyXdI+naiHjjfVfYTY30NSLejYgLJW3Q4CepCxZUZTeNvbe2t0uajIg9Cy2uw5p6LdgaEZskXSbpGtsXv/8SO6mJvi6XtEnSjyNio6T/SJpxb2shsgTUEUlnlb7eIOml2ebYXi7pA5JeHXWs7RUahNMdEXFvI5Xn1khfpxRv5x+RNOMfeVwCmujtVklX2j6swUcwl9i+vYniE2vkORsRU39OSrpPS++jvyb6ekTSkdInKHdrEFjj0/bNu+Lm2nJJL2hwA27qBt75lTnXaPgG3q+K7fM1fAPvBR2/mX+bpJvbvr6e9XWdihuhkk6R9Kik7W1fax96Wzn2s1qavyTRxHN2laTVxZxVkh6XtK3ta+16X4t9j0o6t9j+rqQfjLXuthtXas7lGvym3fOSbizGvifpymJ7paRfa3CDbpekc0rH3lgcd0jSZcXYZzR4G/qkpP3F4/K2r7MHff2EpH1FX5+W9J22r7Evva2svSQDqom+anCv5EDxeGZqzaX2aOL5KulCSbuL14PfSDp9nDXzTx0BAFLKcg8KAIAhBBQAICUCCgCQEgEFAEiJgAIApERAAQBSIqAAACn9H4nGsPaMWuCSAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Visualize the information loss of our sigma\n",
    "interpreter_simple.visualize()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the second and forth words are important to ${\\bf s} = \\Phi({\\bf x})$, which is reasonable because the weights of them are larger."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 How to understand a saved PyTorch model\n",
    "\n",
    "In this section, we will show you how to use our Interpreter in a more complex saved PyTorch model. We use the **3rd layer** of the **pre-trained BERT-base (12 layers) model** for simplicity as an example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Prepare necessary components\n",
    "we first load the pre-trained model we need to explain and define the sentence we use in our case. Suppose the sentence we want to study is `rare bird has more than enough charm to make it memorable.`, and the layer we need to explain is the 3rd layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 231508/231508 [00:00<00:00, 905875.25B/s]\n",
      "100%|██████████| 407873900/407873900 [00:12<00:00, 32166694.25B/s]\n"
     ]
    }
   ],
   "source": [
    "# suppose the sentence is as following\n",
    "text = \"rare bird has more than enough charm to make it memorable.\"\n",
    "\n",
    "# get the tokenized words.\n",
    "cache_dir = TemporaryDirectory().name\n",
    "tokenizer = BertTokenizer.from_pretrained(\"bert-base-uncased\", cache_dir=cache_dir)\n",
    "words = [\"[CLS]\"] + tokenizer.tokenize(text) + [\"[SEP]\"]\n",
    "\n",
    "# load BERT base model\n",
    "model = BertModel.from_pretrained(\"bert-base-uncased\", cache_dir=cache_dir).to(device)\n",
    "for param in model.parameters():\n",
    "    param.requires_grad = False\n",
    "model.eval()\n",
    "\n",
    "# get the x (here we get x by hacking the code in the pytorch_pretrained_bert package)\n",
    "tokenized_ids = tokenizer.convert_tokens_to_ids(words)\n",
    "segment_ids = [0 for _ in range(len(words))]\n",
    "token_tensor = torch.tensor([tokenized_ids], device=device)\n",
    "segment_tensor = torch.tensor([segment_ids], device=device)\n",
    "x_bert = model.embeddings(token_tensor, segment_tensor)[0]\n",
    "\n",
    "# extract the Phi we need to explain, suppose the layer we are interested in is layer 3\n",
    "def generate_BERT_Phi(bert_model: BertModel, layer: int):\n",
    "    assert (\n",
    "        1 <= layer <= 12\n",
    "    ), \"model only have 12 layers, while you want to access layer %d\" % (layer)\n",
    "\n",
    "    def Phi(x):\n",
    "        x = x.unsqueeze(0)\n",
    "        attention_mask = torch.ones(x.shape[:2]).to(x.device)\n",
    "        extended_attention_mask = attention_mask.unsqueeze(1).unsqueeze(2)\n",
    "        extended_attention_mask = extended_attention_mask.to(dtype=torch.float)\n",
    "        extended_attention_mask = (1.0 - extended_attention_mask) * -10000.0\n",
    "        # extract the 3rd layer\n",
    "        model_list = bert_model.encoder.layer[:layer]\n",
    "        hidden_states = x\n",
    "        for layer_module in model_list:\n",
    "            hidden_states = layer_module(hidden_states, extended_attention_mask)\n",
    "        return hidden_states[0]\n",
    "\n",
    "    return Phi\n",
    "\n",
    "\n",
    "Phi_bert = generate_BERT_Phi(model, layer=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Create an Interpreter instance\n",
    "\n",
    "In the following, we'll show you how to calculate the $\\sigma_i$ using functions in this library. To explain a certain $\\bf x$ and certain $\\Phi$, we need to create an Interpreter instance, and pass your $\\bf x$, $\\Phi$ and regularization term (which is the standard variance of the hidden state r.v. $\\bf S$) to it. Here, we use the regularization term we already calculated for simplicity."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.6720064191778069, 0.5696053129989517, 0.5447734704199672, 0.5538335568288567, 0.6839598078248833, 0.6505799523747332, 0.6138378727376542, 0.6199456982157656, 0.6306355030517169, 0.5243086318591497, 0.5000930128511534, 0.4827939168590007, 0.51908702631163, 0.5422031857051318, 0.558117971090347, 0.5384989781528138, 0.4973997679961822, 0.5335898308359087, 0.6593399660006478, 0.6181028809756937, 0.6041363265472766, 0.5372477429375551, 0.598196971464808, 0.6076224088069293, 0.5076913720444645, 0.645040330374062, 0.4911672467247289, 0.6188230685389587, 0.500837794378735, 0.609637156019712, 0.6584791082546065, 0.5070865130054582, 0.5897822361246311, 0.6407670325047011, 0.6215270116703064, 0.5588143832710171, 0.6331066296213183, 0.5231979565110525, 0.5658921746881577, 0.5968348697164344, 0.6109241047763923, 0.6946546444303081, 0.6124973128849484, 0.5661205470990824, 0.6209094314251753, 0.5050536927131517, 0.9198743437277025, 0.6441395669510548, 0.6859488332096185, 0.601743328628138, 0.6657785886030984, 0.5833936191830211, 0.5898142874989561, 0.4939017091306656, 0.553457465711801, 0.5081231009363393, 0.5591332177596285, 0.5049017773625475, 0.6031726394194478, 0.5235285530111075, 0.5822860647045073, 0.49414878949409907, 0.4067730655843068, 0.542062997693116, 0.6219646972540404, 0.5610470463598181, 0.5953315515783884, 0.6029648761076112, 0.559769448508683, 0.6469309658573977, 0.5950178816092214, 0.6448319036619511, 0.6358337907663586, 0.5693825155687869, 0.6652512647877379, 0.46869892954355613, 0.6353106204720417, 0.4562067289029469, 0.4964502603196801, 0.5206233875458052, 0.6428625806381059, 0.6108864631733767, 0.6486324059870165, 0.5352102313528296, 0.4148418325724406, 0.510248378037191, 0.5138279230146225, 0.5603926560966265, 0.595676369003732, 0.640714133095125, 0.537345020757705, 0.5880741669140909, 0.5071738799325218, 0.5429150432027633, 0.5119552619987725, 0.5082208016140594, 0.3948878209321561, 0.5077286211275706, 0.5822582643185853, 0.713473444398512, 0.7043491018182539, 0.6912586136177066, 0.5994933104801177, 0.5322452750878545, 0.46650028309002645, 0.35287523313522456, 0.5493517011722789, 0.6587226734253601, 0.5665924293645145, 0.5068284765701566, 0.6302226696199406, 0.6266167971297549, 0.536432292309199, 0.5948000603725734, 0.6469345665904533, 0.5528489765406325, 0.5280362853934059, 0.6481372026819949, 0.6192433717680882, 0.5450068875551102, 0.5288482106690569, 0.39416141845562686, 0.5245808558547367, 0.5259086532041622, 0.393548948727694, 0.48990314446643507, 0.5703322066306044, 0.44862071323671415, 0.582652673791941, 0.521113021353686, 0.681793501539276, 0.5443441656998707, 0.37084482740748426, 0.6015358235542453, 0.5957221796890386, 0.6995457771130384, 0.5453373563264777, 0.4844465750539164, 0.5715930474350235, 0.5046089569638691, 0.6171893091324396, 0.611591297342313, 0.6827167176256492, 0.7376992908309039, 0.5582270652247779, 0.31228514743652797, 0.6263384555591894, 0.5075020071367358, 0.5834912120567404, 0.5358572981041639, 0.5789452180588497, 0.5361360541953356, 0.45794489247599557, 0.6876262699630906, 0.6650665195124484, 0.5126871568466, 0.6423847524196573, 0.5092494090147971, 0.6336159830721736, 0.3386088037608132, 0.5904231572668198, 0.6248617005511456, 0.5190389568435076, 0.6409503576736226, 0.5527542193297937, 0.6216121274931188, 0.6050308962350159, 0.5212559401366526, 0.5019995204048897, 0.5566706919664209, 0.5758882387663521, 0.6057541264876548, 0.5408059995901172, 0.6402653854107861, 0.6924132203189057, 0.3894033882458203, 0.5251399223179439, 0.5592368166825678, 0.5050009958414926, 0.5484504885732712, 0.6359153476486328, 0.6713817908504806, 0.5528231360245106, 0.7047493918228277, 0.6747710082303218, 0.554946500855997, 0.7382581961284679, 0.5725449044397448, 0.6928065652155618, 0.5869639678552843, 0.5969129932901078, 0.687205195680762, 0.6173225773079768, 0.5625884688571695, 0.554521799472328, 0.599777125422387, 0.6347430361889577, 0.48948299247809623, 0.5300653846822636, 0.47534867458011576, 0.5943453147851882, 0.5653052137388904, 0.3744017641816805, 0.5241624143861529, 0.5400506458238573, 0.5768941593863307, 0.5824398948859419, 0.507294188065184, 0.6074959838986872, 0.5143751868996466, 0.5397642313337914, 0.5562825550742968, 0.5573326668537761, 0.595323709987766, 0.5335037599947999, 0.40645409459315884, 0.38584858519561777, 0.6763269336685823, 0.49864392338741487, 0.5556405102765481, 0.56796873872809, 0.6704090264129464, 0.6365633549192408, 0.491858674581239, 0.571182550422225, 0.39636775642315364, 0.4927436956522672, 0.5325300657983706, 0.5914024175418211, 0.6572771291592268, 0.5704864304921875, 0.595574678005175, 0.5556762243555338, 0.6367940265899822, 0.588938002385068, 0.5494135473896908, 0.5194181997174186, 0.4797671520725825, 0.6518392146394686, 0.46972510885465957, 0.5284412162612128, 0.5815074512359419, 0.5823763772131357, 0.5029126283299797, 0.5109125789183494, 0.3649196005975494, 0.4906738472864752, 0.5488590983504192, 0.49753811165261946, 0.505679716613382, 0.5129037523367018, 0.6311914470548811, 0.5946013817765508, 0.6477204415156932, 0.6515995975676336, 0.5768488525143426, 0.523685718723664, 0.5295288664585905, 0.5009731898317803, 0.5587233774569005, 0.6580833925637624, 0.5373233954142419, 0.6582209087756435, 0.5489848637461159, 0.47337027482398153, 0.5043031106374255, 0.577317464046264, 0.5219168721158063, 0.5408242665558518, 0.6356068922290092, 0.5733903517119527, 0.6016504227165926, 0.5288946338379021, 0.5892373351065394, 0.6342615103769079, 0.6130451086636514, 0.46611514729384973, 0.5921430847081095, 0.5779202188955178, 0.6357414621588608, 0.5030608331783005, 0.6395649333620499, 0.6377276982623341, 0.38973320353067653, 0.606781050411065, 0.5093703899551436, 0.5811707770744574, 0.7232476284557209, 0.5168817343337841, 0.6638206576529871, 0.48901418733542623, 0.43739760969575475, 0.6015852419136676, 0.6279423808477765, 0.5814834444420861, 0.4437173666943161, 0.415649335926807, 0.5494089646655524, 0.5699369266474212, 0.5716393111700822, 0.7203262934280066, 0.6109806114151356, 0.6086680149215294, 0.6784207314117051, 0.7510138193211784, 0.595752370855507, 0.5096256983929184, 0.6509022221259164, 2.5251604297907906, 0.5996432671274571, 0.5191965465959889, 0.3910454371052007, 0.5252704538224279, 0.5487563320085694, 0.6497572197588998, 0.539947818113331, 0.5727221067381307, 0.5859855369201918, 0.5550066573682404, 0.4052812429107734, 0.5767050515446769, 0.5074083423324552, 0.5181425538650056, 0.5867669850438991, 0.622336928241333, 0.619730307042242, 0.5159150588135848, 0.5677132852737771, 0.6088984959356519, 0.55785345177934, 0.4887743221019428, 0.4488903808698761, 0.5784225594460584, 0.6405666164702244, 0.6272819335447044, 0.5812181962692605, 0.5423161572251998, 0.6681405021562552, 0.5763012190181668, 0.5985999991742619, 0.48662936326087963, 0.8344473533595789, 0.4856559120241086, 0.5897134269617805, 0.5323088631490295, 0.4448512715743428, 0.5975625673827604, 0.5743464047665394, 0.5417000727311233, 0.6900951864217544, 0.5287838019788256, 0.6675197469611557, 0.6185928209906237, 0.35727423668025304, 0.6143233750515678, 0.6060399652428077, 0.5558822642469178, 0.530276102166812, 0.5116897130040144, 0.5977306063905552, 0.5867864648924253, 0.5278337771927558, 0.4806879149051836, 0.5382478622650878, 0.5856067571861844, 0.6306318442056607, 0.49391258821445383, 0.5489518148259469, 0.6131126056440158, 0.5089564625233208, 0.6258332685683875, 0.5562664332792977, 0.6587553832219688, 0.5721395069393773, 0.7316276447951, 0.6815503509226922, 0.6413576967825804, 0.5484064973484217, 0.5643773131034424, 0.6745771327685542, 0.585569843073139, 4.148380916210813, 0.5982397003506298, 0.7019254715223583, 0.5288198822451418, 0.6240095756994913, 0.5380081185581451, 0.678795541845524, 0.588529980887846, 0.6280178517136987, 0.4657859039190076, 0.5132997470267544, 0.5313934116466171, 0.5266762597476911, 0.6660931196796194, 0.7234808055126262, 0.594061226970833, 0.5515582546731305, 0.5515454825862416, 0.6217069320805647, 0.542655348574746, 0.44244111415482446, 0.462192639977372, 0.5957314889234636, 0.4433120212592345, 0.5494358069465771, 0.6681352256096942, 0.5277128695902392, 0.6099108527298399, 0.6512378710022276, 0.5679475425471902, 0.5546903245945867, 0.6674891410259085, 0.6159889738277975, 0.5735603517089791, 0.5248157975023054, 0.5168292372862809, 0.553843050306285, 0.5985645439688244, 0.39715573701478935, 0.5283503609143612, 0.447520945026398, 0.638239313062452, 0.507419952536012, 0.5760147366093716, 0.5515266316019759, 0.5807239816994251, 0.5599335674825909, 0.5974892666780456, 0.5004662990922536, 0.5888220193638405, 0.4718941619780529, 0.5861252978337973, 0.49223294458949224, 0.6064087992507717, 0.7214565901405107, 0.5484770327034333, 0.5503630111171568, 0.5904485777568088, 0.5939656911153031, 0.5038700884054164, 0.6045103501937796, 0.499118136236574, 0.37193186488077934, 0.5723301608055107, 0.525825703005349, 0.7301040665136092, 0.5140891502639778, 0.5638728483758652, 0.4977489920384463, 0.6471063133748627, 0.6190685639017889, 0.6775629213383808, 0.612278641490936, 0.767169815870711, 0.6605883129662021, 0.6187656212507987, 0.6353017504729984, 0.5077724881945187, 0.6115224371318742, 0.5697210817105954, 0.45587282019630315, 0.6138376361062206, 0.6027143188401093, 0.5812920517773535, 0.5187384796774935, 0.4845791539559439, 0.5139248717515621, 0.5097321564832612, 0.5702922031533869, 0.7662102767707549, 0.5100037970767839, 0.5769086276467204, 0.5918487388928512, 0.4932610414911433, 0.5612912007434486, 0.5019870022881627, 0.5683701006907799, 0.5471292375834348, 0.5163464077571903, 0.5662265235043846, 0.6262520734546206, 0.4045579543142629, 0.5740986318745384, 0.5404645788548182, 0.5729629704291168, 0.543927143587405, 0.6151617140048048, 0.5022090234235793, 0.5783303424697744, 0.616379942383976, 0.5646497198031761, 0.5213611237979402, 0.5846124785989514, 0.6169145129780825, 0.534197561211705, 0.4465214795463175, 0.5524803477799444, 0.6330414104074786, 0.49492489969592696, 0.5329551473534944, 0.5929020635462238, 0.5238675494097854, 0.5429900230821683, 0.5178870028057245, 0.6226470853153954, 0.5495123138896764, 0.5208366967349471, 0.6039583328156696, 0.5546041461090794, 0.47913038384104656, 0.5260999213211526, 0.5471560408038963, 0.5978561074109934, 0.5814516514009886, 0.6079996257427562, 0.634367589709343, 0.36981740415466857, 0.6970604594957829, 0.5697476683491814, 0.5939516289500889, 0.49108811101156746, 0.6614081220367268, 0.5676172157021177, 0.5633578995510955, 0.6006402768683331, 0.5588224559173837, 0.6092733345232592, 0.5769390725536546, 0.37429565713285773, 0.5444970698964151, 0.4640715629842989, 0.6700378566201302, 0.6034207436950111, 0.6619094031660634, 0.5034445564834289, 0.5866391165503538, 0.6139727787785322, 0.4639254512909701, 0.8315876916202657, 0.475982003413719, 0.4372537729306341, 0.5722447067075571, 0.5882645819759047, 0.5779057015430793, 0.6627630369268039, 0.5916307899922117, 0.5342299317908815, 0.5362378111872667, 0.5442251792617785, 0.6348450888774665, 0.4871357148469115, 0.5614704400230816, 0.6329994610904498, 0.6671062619076585, 0.5660308869774, 0.553753223172401, 0.5062704350286694, 0.6968807651805093, 0.6275035575642098, 0.6714519055052968, 0.5595083618517822, 0.5262363525655602, 0.6077647308803884, 0.5445943891364097, 0.6106190753878226, 0.5087365182910752, 0.6428060786938798, 0.5572595189411257, 0.5982255767281885, 0.6064772943030274, 0.641796870858977, 0.5625580596207618, 0.4024176418766817, 0.5646488254069216, 0.5803300611004657, 0.6054487459605079, 0.39705800608168873, 0.6481802283924225, 0.6511188758003585, 0.5391900380698975, 0.5814531439531853, 0.6471624238647458, 0.49907443129699103, 0.5519686262012745, 0.5868227143748256, 0.6935372558308686, 0.6449801314172242, 0.49368417061492087, 0.6030356697914756, 0.655457963578686, 0.6140634241247342, 0.7083376470121734, 0.5302490023499564, 0.5004156026909182, 0.5294756142889654, 0.5752491194188754, 0.5515118322540393, 0.6266358404603671, 0.5721215724933211, 0.5709573199551158, 0.6895899657142461, 0.6884385663549084, 0.5552502177229434, 0.5993480716206604, 0.511304209248702, 0.7221669262906404, 0.5022257278527431, 0.5905228187869876, 0.5923767227511361, 0.657322623740696, 0.5205411847873718, 0.6163160251297007, 0.4961048543439325, 0.45154069182902334, 0.5853784934736037, 0.47034049733531025, 0.7105284045363544, 0.5979161016416713, 0.49215242740699805, 0.5667098695380436, 0.5943849976101567, 0.7154269400608548, 0.583378099295273, 0.5835240865444338, 0.6486352085980603, 0.6384153780423892, 0.5384197352251137, 0.5448564471991062, 0.6488850948552147, 0.5294210616757474, 0.517439144119657, 0.5270373136055498, 0.5616827422722539, 0.5868951702028061, 0.5565920647445286, 0.6232391803921958, 0.6020491794954504, 0.46992208568892646, 0.6486807312726811, 0.5613844933224755, 0.555391749084397, 0.5772819796113084, 0.45810557230990834, 0.6398638599377786, 0.5366702136482702, 0.5739951459427881, 0.5982143252184688, 0.5457185094619164, 0.5665062960826616, 0.7118753111945728, 0.5854817189813198, 0.5800307675409715, 0.5544727708631073, 0.3338893382006267, 0.5091045621758792, 0.46165716062730827, 0.6030307388526068, 0.6080089650119052, 0.5194952380136443, 0.6433003128528647, 0.6914219001844641, 0.6480064308291791, 0.7222649910184369, 0.5560988035902191, 0.5104647434087874, 0.679415076191899, 0.5328008354866952, 0.5899761539786094, 0.5390000973833181, 0.3701084043163154, 0.6339217411098885, 0.593559378938782, 0.5144669461423722, 0.5881084008662535, 0.551622833665805, 0.571227894472472, 0.6411525062691923, 0.7151859458336626, 0.6522025417511709, 0.6322082840004087, 0.6860187619791485, 0.626165217142063, 0.5916004662153633, 0.6487841124590062, 0.5309168041739798, 0.48527289381010597, 0.6020822653764765, 0.6807790880919588, 0.5898577639409405, 0.6344649343044279, 0.6173375657307242, 0.5890823347628427, 0.6726218694450057, 0.5859075938014776, 0.5815875066102324, 0.5758172547583169, 0.48712524314796934, 0.5377782629361193, 0.3938382744806918, 0.676308704853732, 0.6677226491785414, 0.5223299611243529, 0.5495415131548628, 0.6108127620421834, 0.4988800373920646, 0.5072853471520158, 0.5324874634673668, 0.6421144110599337, 0.5641352537195086, 0.4815342678555669, 0.5920600911108078, 0.500374587993995, 0.5367396038113736, 0.5451746772498243, 0.6622860029957304, 0.5333077496098986, 0.5155271033413926, 0.607286350062922, 0.5020017637448351, 0.516500845429085, 0.629443864762706, 0.7105638158452252, 0.5329168640353985, 0.5608956952115493, 0.6469797564951193, 0.5398440588271034, 0.5556897712098293, 0.6163183363303857, 0.610403496455368, 0.5996230941159092, 0.6347018234826138, 0.6549477032938691, 0.6329173885830707, 0.4950253371618349, 0.5230316699129772, 0.5882612702166155, 0.5079731724738321, 0.6786573152179876, 0.46707104070019356, 0.5800420961252108, 0.6110073061061204, 0.7033818294980708, 0.6022168812302134, 0.5288295241084625, 0.5626981078946942, 0.543953278377877, 0.6310410566957821, 0.5937899619372705, 0.6103545565256665, 0.660002389861427, 0.6041752509423339, 0.5308771948309202, 0.572450204884434, 0.5689950110928802, 0.48266978637745767, 0.5245673168231065, 0.6406084034155108, 0.52956038481703, 0.4625944334907985, 0.5517903459951017, 0.5861719638693285, 0.5376046382766501, 0.5671392388053157, 0.5736645873541928, 0.46055001581443344, 0.5001459012929047, 0.7395769297862513]\n"
     ]
    }
   ],
   "source": [
    "# here, we load the regularization we already calculated for simplicity\n",
    "data = request.urlopen(\"https://nlpbp.blob.core.windows.net/data/regular.json\").read()\n",
    "regularization_bert = json.loads(data)\n",
    "print(regularization_bert)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "interpreter_bert = Interpreter(\n",
    "    x=x_bert, Phi=Phi_bert, regularization=regularization_bert, words=words\n",
    ").to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Train the Interpreter\n",
    "\n",
    "Then, we need to train our interpreter (by minimizing the loss [here](#0.2-Perturbation-based-Approximation)) to let it find the information loss in each input word ${\\bf x}_i$ when they reach hidden state $\\bf s$. You can control the iteration and learning rate when training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 5000/5000 [00:58<00:00, 85.92it/s]\n"
     ]
    }
   ],
   "source": [
    "interpreter_bert.optimize(iteration=5000, lr=0.01, show_progress=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 Show and visualize the results\n",
    "\n",
    "After training, we can show the sigma (directly speaking, it is the range that every word can change without changing $\\bf s$ too much) we have got. Sigma somewhat stands for the information loss of word ${\\bf x}_i$ when it reaches $\\bf s$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.17860198, 0.14068164, 0.15262878, 0.22471362, 0.20457381,\n",
       "       0.21281476, 0.18869533, 0.13970219, 0.25510186, 0.22200805,\n",
       "       0.24051382, 0.1302286 , 0.2824908 , 0.36167043], dtype=float32)"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sigma_bert = interpreter_bert.get_sigma()\n",
    "sigma_bert"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAacAAAB4CAYAAABW8D3WAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjAsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+17YcXAAATt0lEQVR4nO3de7gdVXnH8e8vF0IAyYUgAhESeFAbLGAJIAoSWiqohWhBqIJNvFFslYpFHy0WI4it0Kr1WtSnDSoWRCsFtFwEAoJGTCB3iCQkCKJoEAOYACF5+8daO2fOzuyz9z5nn32Gk9/nefZz1p5ZM++aNWvmncsOKCIwMzOrkhFD3QAzM7N6Tk5mZlY5Tk5mZlY5Tk5mZlY5Tk5mZlY5Tk5mZlY5o9qpPHKncTFq/B6D1ZZSmzZs7Go8AEa11S0dMWHi2K7HfHZT9/8ZwYiR6nrMZ5/Z3PWYzzy5oesx99p7p67H3LCp+8fK6JFbuh7ztw8/0fWY+03duesxx+3Q3ZgPrv0F69Y9VnpSaGtkjRq/B3ue+bnOtKpFj959X1fjATBhYtdDHn/6QV2P+cijm7oec+zOo7se86E1j3c95up5i7oe8+yLXtH1mAsemdD1mJPHdf+C9cv/eGPXY3567iu7HvO1k4/oarxXHzGj4Tw/1jMzs8pxcjIzs8pxcjIzs8pxcjIzs8pxcjIzs8pxcjIzs8pxcjIzs8pxcjIzs8pxcjIzs8pxcjIzs8pxcjIzs8pxcjIzs8ppmpwknSlpgaQFmzes70abzMxsO9c0OUXEVyJiekRMH7nTuG60yczMtnN+rGdmZpXj5GRmZpXj5GRmZpXj5GRmZpXj5GRmZpXj5GRmZpXj5GRmZpXj5GRmZpXj5GRmZpXj5GRmZpXj5GRmZpXj5GRmZpXj5GRmZpWjiGi9svRb4MF+xJkErOvHcgPhmI7pmI7pmNWOuW9E7F42o63k1F+SFkTE9EEP5JiO6ZiO6ZjDIqYf65mZWeU4OZmZWeV0Kzl9pUtxHNMxHdMxHXMYxOzKOyczM7N2+LGemZlVjpNTiyRNkbSsZPrXJE1rYfkZkq7rVNzhQNJ4SX+by/3qn6qRtFbSpJLpcyWdMhRtqmvH1j6vAkmzJX1hkGP8OP+dIumtgxmrEyTNkXRuyfRhey4o06/klDtpo6RF+fuLJF0habWkFZJ+IOklfZzQXynpp5IWSbpX0pw8/TRJq9o5SSlpuh2SRraxiS2LiHdFxIpuxasySaPaXGQ8UJkTZVW1OsZbtN31eUS8KhenAJVITv04Vrqq5Bx/nqTlkpbk8/YRefo8SSvztEWSvpOnz5H0yzxtmaST8vRzJP2ipQuSiGj7Q9rJy3JZwE+AswrzDwGOLtarW34lcHAujwSmFebNAK5rIf69wJeAe4D/AhYAy4GPF+qtBc4H7gD+CtgfuB5YCPwIeFmb23wfcBmwBPgOsBMwD5ie6zwFXAD8FDgKOCEvcwfwuWbb1WRbv5q370ZgLPBu4GfAYuC7wE65/puBZXn67W1u29fyspcDxwF3AvcDhwMTgavzts8HDsrLziG9DL0R+Fben5fkti0B/qaPuFcAG4FFuf683K/35TbU3omen+cvy7Fq0+cBnwLuAn4OHF1Y9xl5+iLg0tyup4CLct/MB/bIdfcFbs7tvRnYJ0+fC5xSWOdT+e8I0thbDlwH/KBWjzTmPp7/bszb8o28rs8BPwYeKNTfJce8G1gKzGwwxvfN7f8Uafz+MO+XeXl9J7W4r4t9fkn+LMuxTxvA+aDZ+Dk8b/s9+e9L87KzgS/k8htI55JJwO6kcf2z/Hl1f9pWt9/mA08AT+dyo7buDPxnjntPYZ/MJh0D1wJrgPcCH8h15gMTC+e/+Xk8fQ+YUBivnwRuA/4BOJF0rrgn78/aeJyTx8wtuU3vLjnvtnycDWCf1mIdmffLmPx9ErBXYZumlyw/Bzg3l/+I9A90R9Tv8z7b0IGG/ykNToI0Tk6PAy9ssMwMWktOW4BX5u+1QTEyd1btxLkW+FBhuZuBA3L5COCWNrc5yAdJHrzn0js5BXBqLu8IPAQcQErg3262XX3EfQ44JH//NunEu1uhzieA9+XyUmDvXB7fZow/Jp14F+btEzCTdEB+HvhYYZ8vKgzChcDY/P1M4KO5PIZ00TC1hXE0A1gPTM5t+AlwVHH/5vI3gBMLB8a/5fLrgR8WDoZrgdH5+5eAv877p7bsxYV2XgvMyuV3AFfn8lzKk9MppIQ0AngRaTwXk9NFpAuwD5JO2BPzuq7Ky0wDVuX6o4BdCwf9qtzvUyiM8cL4el0uf490QTAaOLi2P9o8dk8GbiIdN3sAvwD2HMAY7Wv87AqMyvWPA76by7OBLwBvIl0w1k7k3yrs/32Ae9ttV8l+m0E6BzRr6yeBM2rHEOnCZ+fc1lXAC0jJcz35ohz4DPD+XF4CHJPLFwCfLYzXLxXaNYGeC6130TOW55AuoMbmMfEQsFfdvmv5OOtnnxVj/SVwbYN682iSnPL3X5PP+bSYnDpxa/ly0g5ux2eAlZLmke5kLouIp9tcx4MRMT+XT5V0JulA35N08C/J864EkLQL8CrgKkm1dYxpM+ZDEXFnLn8TOLtu/mbS1R7Ay4A1EXF/jv9N0oDqjzURsSiXF5IGzsslfYJ08OwC3JDn3wnMlfRt4H/ajLE0t3U5cHNEhKSlOd6+pJMZEXGLpN0kjcvLXhMRG3P5tcBBhfcr40gJek0LbbgrIh7ObViU494BHCvpQ6Q71YmkO5Zr8zK1baz1C8CfAYcCP8v7eizwG+BZ0p1Orf6f5/KRpAMQUvK7uEk7jwKuiogtwK8l3Vo3fyPpDvB24PiI+F1ux9V5mRWS9sh1BXxS0mtIyWhvUqKA3mOc3P7rc3kp8ExEbCrso3YdBfx3RGwGHpV0G3AYcE0/1tVs/IwDLpN0ACnJji4seywwHXhtRDyRpx0HTCscq7tKekFEPNmPtrXb1snASYX3PjuSEiTArbkNT0paT884XEoa9+NIF4W35emXkS5Kaq4slCcDV0raE9iB3sfI/+ZjamMeX4eT7nZrBnKctetG4HxJPyfd4V1Z2D6AyyXVjv+bIuKDxYXzI8AtwG/bCTokzz0j4gJJl5M6+K3AW0hXNe34A4CkqaQ7mMMi4nFJc0mDqVc90lXS7yPikIE0vcn3p/OB3mh+fz1TKG8mnWznAm+MiMWSZpP7LyLOyoPhDcAiSYdExGNtxthS+L6FNE6eK1mmtn1/KEwT6S7uhpL67bRhMzBK0o6kO5/pEfFQfj+5Y8kym+kZzyJd8HykuHJJ50a+dKurX69W5znye1mls+QOhfX3ZXNeR32M4vbV1nE66Sr80Jxo1tKzfcV+BdhUaP/WfRQRW/r5DqPZdrSj2fi5kHRif5OkKaQr7poHgP2Al5DuACD1+5GFi55OatbWzcDJEbGyuFA+rpot20xxn34e+HREXCNpBuluo6bZuWYgx1lbIuIpSYeSXtUcS0qoH46IubnK6RGxoGTRcySdATxJemTc1vmwEy9Zl5OuUtsSEasj4sukq9yDJe3Wz/i7knb4+nw1+roG8Z4A1kh6M2x9yXxwm7H2kXRkLr+FdFXfyH3AVEn7F+p30guAX0kaTTrBASBp/4j4aUScT3rO++IOxbu9FicfSOsKV7lFNwDvye0i/zBm5wbrfDJvR19qJ+p1+e63lV+83QycIumFuQ0TJe3bR/0fk95JQtrG2n5dS8/YnknP1f4dwMmSRuQxN6NufbcDp5KuZpE0sY/Y44Df5MR0LOkOdTAV+/x24DRJIyXtDryG9J5uMIwDfpnLs+vmPUi6c/26pAPztBtJ73QAkDSQi8qaJ0mP55q5AXhfviBB0itaDRAR64HHJR2dJ72N9I6pTLFPZtXNmylpx3xenEF6t1TfxlaPswGLiM0RMS8iPkbaLye3sNhnIuKQiDg6In7UbsxOJKdbgDGS3l2bIOkwScc0WkDSG2o7nnQruhn4fX+CR8Ri0gvF5aRnx3f2Uf104J2SFuf6M9sMdy8wS9IS0uOlL/fRrqdJj/G+L+kO+vdfc+/LP5Fept5ESoQ1l0haqvQrydtJz647YQ4wPW/7v7DtwVTzNWAFcHduw6U0uKLMd3R35nqXNKjze9KPQZaS3gfUH6Rly6wAPgrcmNt7E+lxbyNnA2/Pdd8G/H2e/lXgGEl3kd5R1q56vws8THqhfilpP6wvrG8l6b3TpaTHVZ/uI/blpH5dQBqf9/VRd8Dq+vxI0uPvxaTj+EMR8etBCn0x8M+S7iS946pv10rS9l+VL+jOJo83SSuAszrQhiWkc83+ks7po96FpAuRJbmfLmwzzizScbiE9OOICxrUm0Pa3h+x7X/R+y7g+6QfVlwYEY/UzS89zpR+Kb1Xm+3tk6SX5sexNYfQ+fPZtnHbvNNKC6Xb8usi4uX5+17AZ0lXmU+TrjjfD2wi/drk0cLi55Cy7p8AG0iPTs6r3Z7mq/JzI+Iv+rE9Zl0haZf8uGM30onk1YN4YjfrquI5Pj/S+zzp/fZzpB+FnBkR65R+N7An6T0rpCcqx+XH709FxL+WrHs26TH9e+vnFXXknVPO6qc2mD26ZNpVJdPMnk+ukzSe9B7qQicmG64iYiHpx2Rl82Y0mD5noHH7+1hvMzBO+R9odYqk00gvvx/v5HrNOi0iZuTn6dMKL4bNhovBOsefA3yE9O/N+q7bn8d6ZmZmg8n/bT0zM6scJyczM6scJyczM6scJyczM6scJyczM6scJyczM6scJyczM6scJyczM6scJyczM6scJyczM6scJyczM6scJyczM6scJyczM6scJyczM6scJyczM6scJyczM6scJyczM6scJyczM6scJyczM6scJyczM6scJyczM6ucUe1UHjnxgIjnNpTMEahsibLpKi1u/dKoflndXsWSBqj59KarLZ2v1jerpI2NYpY0bwDLaduqpRsFalBBDerXZvbVR9v2gcqqbfOlbJ2Nd2OUNatU+VDbdvnGdRutN7ap0LhuyTS10Iayeaqf1vp66vuo52tJfzZYuJ1t7D2/YfAWli2v1dI+b7L28rl9N66d9feu1UfdbbqntfW2vP76us1ODqVzm/ZW33MLaeHuhYtuiIgTypZoKznFpg2MOewsGJFvuGqjQgLlaSNUmD6ipzyiZJpGFKY3WEaFWFunl0xrc12qHWRbq6qwuAqbptJpPavqWc+IRnULMXqaXVZWoamt1VVd3RF9zq/fhro+kArL915um/gj6rexp+0988uXK9uWnr6Pwi5LJ0sVy8Vh0HB+KtceC0jRa/1by5TEUu/1ls6nJFajuIVy7YBs2O7C/LK6xfW3E2vrNtDfWCpM7336EyqsvzAmaqdJlZd72tpz+m1Yri3TYF3FmMVyMUatxWowbWu0XtunresqXsCpbl3pQq2nbs/k+i3IJZXE7bUthbh1V0f10+rjqtc2qEEbi20pWW+h/c3WW+y3RtvYV7vHjho/iQb8WM/MzCrHycnMzCrHycnMzCrHycnMzCrHycnMzCrHycnMzCrHycnMzCrHycnMzCrHycnMzCrHycnMzCrHycnMzCrHycnMzCrHycnMzCrHycnMzCrHycnMzCrHycnMzCrHycnMzCrHycnMzCpHEdF6Zel6oOH/VreLJgHrhroRw5j7d3C5fweP+3Zwdbp/10XECWUz2kpOVSFpQURMH+p2DFfu38Hl/h087tvB1c3+9WM9MzOrHCcnMzOrnOdrcvrKUDdgmHP/Di737+Bx3w6urvXv8/Kdk5mZDW/P1zsnMzMbxiqXnCSdIGmlpFWSPlwy/zWS7pb0nKRT6uZtlrQof67pXqufP1ro3w9IWiFpiaSbJe1bmDdL0v35M6u7La++Afatx24TLfTvWZKW5j68Q9K0wryP5OVWSjq+uy2vvv72raQpkjYWxu5/dKxREVGZDzASWA3sB+wALAam1dWZAhwEfB04pW7eU0O9DVX+tNi/xwI75fJ7gCtzeSLwQP47IZcnDPU2VeUzkL7N3z12B96/uxbKJwHX5/K0XH8MMDWvZ+RQb1NVPgPs2ynAssFoV9XunA4HVkXEAxHxLHAFMLNYISLWRsQSYMtQNPB5rpX+vTUiNuSv84HJuXw8cFNE/C4iHgduAkr/8dx2aiB9a8210r9PFL7uDNReqM8EroiIZyJiDbAqr8+SgfTtoKlactobeKjw/eE8rVU7Slogab6kN3a2acNCu/37TuD/+rns9mYgfQseu8201L+S/k7SauBi4Ox2lt2ODaRvAaZKukfSbZKO7lSjRnVqRR2ikmntZOh9IuIRSfsBt0haGhGrO9S24aDl/pV0BjAdOKbdZbdTA+lb8NhtpqX+jYgvAl+U9Fbgo8CsVpfdjg2kb39FGruPSToUuFrSgXV3Wv1StTunh4EXF75PBh5pdeGIeCT/fQCYB7yik40bBlrqX0nHAecBJ0XEM+0sux0bSN967DbX7vi7AqjdgXrs9q3ffZsflT6WywtJ765e0pFWDfXLuLqXbqNIL9qn0vNi7sAGdedS+EEE6SX9mFyeBNxP3Uu97f3TSv+SToqrgQPqpk8E1uR+npDLE4d6m6ryGWDfeux2pn8PKJRPBBbk8oH0/kHEA/gHEZ3q291rfUn6QcUvO3VeGPKOKemo1wM/zwfxeXnaBaQrTYDDSJn+D8BjwPI8/VXA0tyxS4F3DvW2VPHTQv/+EHgUWJQ/1xSWfQfpZfIq4O1DvS1V+/S3bz12O9a//w4sz317a/EES7pbXQ2sBF431NtStU9/+xY4OU9fDNwNnNipNvm/EGFmZpVTtXdOZmZmTk5mZlY9Tk5mZlY5Tk5mZlY5Tk5mZlY5Tk5mZlY5Tk5mZlY5Tk5mZlY5/w+/x818iK0BrQAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "interpreter_bert.visualize()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the word 'rare', 'bird', 'charm', 'memorable' is important to the third layer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/scrapbook.scrap.json+json": {
       "data": [
        0.003160591935738921,
        0.0015862112632021308,
        0.00629778765141964,
        0.0015863593434914947,
        0.003082597628235817
       ],
       "encoder": "json",
       "name": "sigma_numbers",
       "version": 1
      }
     },
     "metadata": {
      "scrapbook": {
       "data": true,
       "display": false,
       "name": "sigma_numbers"
      }
     },
     "output_type": "display_data"
    },
    {
     "data": {
      "application/scrapbook.scrap.json+json": {
       "data": [
        0.17860198020935059,
        0.14068163931369781,
        0.15262877941131592,
        0.22471362352371216,
        0.20457381010055542,
        0.21281476318836212,
        0.18869532644748688,
        0.1397021859884262,
        0.25510185956954956,
        0.22200804948806763,
        0.24051381647586823,
        0.13022859394550323,
        0.2824907898902893,
        0.36167043447494507
       ],
       "encoder": "json",
       "name": "sigma_bert",
       "version": 1
      }
     },
     "metadata": {
      "scrapbook": {
       "data": true,
       "display": false,
       "name": "sigma_bert"
      }
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# for testing\n",
    "sb.glue(\"sigma_numbers\", list(sigma_numbers))\n",
    "sb.glue(\"sigma_bert\", list(sigma_bert))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}