CS410_Explain.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "CS410 Explain",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "metadata": {
        "id": "aqU6WmMIGw_H",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Note\n",
        "\n",
        "This notebook demonstrates training model to explain slang with context with the following steps:\n",
        "\n",
        "\n",
        "1.   Loading .csv data which contains target word, example of the word in sentences, and the explanation of the word. While loading, it constructs languages for input and output.\n",
        "2.  Set up encoders, combiner, and decoder\n",
        "3.  Model Training\n",
        "4.  Evaluating model\n",
        "\n",
        "We developed this notebook to be run in Google Colab as it offers free GPU for upto 12 hours. Please make sure GPU is enabled in Edit - Notebook Settings. There are also some setup to be run in the beginning including installing pytorch and connecting to Drive.\n"
      ]
    },
    {
      "metadata": {
        "id": "plcZWRRpKiKk",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Colab Setup"
      ]
    },
    {
      "metadata": {
        "id": "QIvFD5HiAnPZ",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Install pyTorch and Download nltk resource file\n",
        "\n",
        "In order to run this notebook in Google Colab, we must install pytorch in the instance running the notebook."
      ]
    },
    {
      "metadata": {
        "id": "QyEnTUNpmjkm",
        "colab_type": "code",
        "outputId": "8ccc9382-a0c0-4208-f4a3-d9cc3a032f62",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 122
        }
      },
      "cell_type": "code",
      "source": [
        "!pip3 install torch torchvision -U"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Requirement already up-to-date: torch in /usr/local/lib/python3.6/dist-packages (1.0.0)\n",
            "Requirement already up-to-date: torchvision in /usr/local/lib/python3.6/dist-packages (0.2.1)\n",
            "Requirement already satisfied, skipping upgrade: six in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.11.0)\n",
            "Requirement already satisfied, skipping upgrade: numpy in /usr/local/lib/python3.6/dist-packages (from torchvision) (1.14.6)\n",
            "Requirement already satisfied, skipping upgrade: pillow>=4.1.1 in /usr/local/lib/python3.6/dist-packages (from torchvision) (5.3.0)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "G25GZyCTmPeY",
        "colab_type": "code",
        "outputId": "62be1f6e-88fb-49e6-a642-071f4bd5c7ec",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "cell_type": "code",
      "source": [
        "from __future__ import unicode_literals, print_function, division\n",
        "from io import open\n",
        "import unicodedata\n",
        "import string\n",
        "import re\n",
        "import random\n",
        "\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "from torch import optim\n",
        "import torch.nn.functional as F\n",
        "\n",
        "import nltk\n",
        "nltk.download('averaged_perceptron_tagger')\n",
        "\n",
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
            "[nltk_data]     /root/nltk_data...\n",
            "[nltk_data]   Package averaged_perceptron_tagger is already up-to-\n",
            "[nltk_data]       date!\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "QPQnW37vAw_B",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Verifiy if CUDA is enabled"
      ]
    },
    {
      "metadata": {
        "id": "8F4EYwBd1Fkk",
        "colab_type": "code",
        "outputId": "64fdc78c-be58-4b7a-b37c-4158f1ea443a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "device"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "device(type='cuda')"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 19
        }
      ]
    },
    {
      "metadata": {
        "id": "Om_MU0nWA5mq",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Mount Google Drive for data\n",
        "\n",
        "- Data file needs to be placed in Google Drive to be loaded in Colab environment. \n",
        "- Also, the Google Drive space will be used to store and reload the model.\n",
        "- Override DRIVE_PATH to the directory that contains the data.\n"
      ]
    },
    {
      "metadata": {
        "id": "anG6ksfIVCRU",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import os\n",
        "\n",
        "MOUNT_PATH = '/content/drive/'\n",
        "DRIVE_PATH = \"My Drive/cs410-explain\"\n",
        "LOCAL_PATH = os.path.join(MOUNT_PATH, DRIVE_PATH)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "k8YYu8Lqm20G",
        "colab_type": "code",
        "outputId": "f08bfa2e-a7f6-4504-b67a-d1fbe1778765",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        }
      },
      "cell_type": "code",
      "source": [
        "from google.colab import drive\n",
        "drive.mount(MOUNT_PATH)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Drive already mounted at /content/drive/; to attempt to forcibly remount, call drive.mount(\"/content/drive/\", force_remount=True).\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "-LLh6bhQok9D",
        "colab_type": "code",
        "outputId": "8b3102fd-218a-485f-e9f1-14947856245c",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "!ls LOCAL_PATH"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "ls: cannot access 'LOCAL_PATH': No such file or directory\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "bgr53rrlERzF",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Data Loading"
      ]
    },
    {
      "metadata": {
        "id": "VP0Y0zgSKpsr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Language Definition\n",
        "The language here contains informations such as word index, character index, POS tag index to encode the input word, character, or tag in dense vector. "
      ]
    },
    {
      "metadata": {
        "id": "HRN1wwSJmPea",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "SOS_token = 0\n",
        "EOS_token = 1\n",
        "\n",
        "\n",
        "class Lang:\n",
        "    def __init__(self, name):\n",
        "        self.name = name\n",
        "        self.word2index = {}\n",
        "        self.word2count = {}\n",
        "        self.index2word = {0: \"SOS\", 1: \"EOS\"}\n",
        "        self.n_words = 2  # Count SOS and EOS\n",
        "        self.char2index = {}\n",
        "        self.char2count = {}\n",
        "        self.index2char = {0: \"SOS\", 1: \"EOS\"}\n",
        "        self.n_chars = 2\n",
        "        self.tag2index = {}\n",
        "        self.n_tags = 2\n",
        "\n",
        "    def addSentence(self, sentence):\n",
        "        sentence = sentence.strip()\n",
        "        for word in sentence.split(' '):\n",
        "            self.addWord(word)\n",
        "        for char in sentence:\n",
        "            if char not in self.char2index:\n",
        "                self.char2index[char] = self.n_chars\n",
        "                self.char2count[char] = 1\n",
        "                self.index2char[self.n_chars] = char\n",
        "                self.n_chars += 1\n",
        "            else:\n",
        "                self.char2count[char] += 1\n",
        "        for word, tag in nltk.pos_tag(sentence.split(' ')):\n",
        "          if tag not in self.tag2index:\n",
        "            self.tag2index[tag] = self.n_tags\n",
        "            self.n_tags += 1\n",
        "\n",
        "    def addWord(self, word):\n",
        "        if word not in self.word2index:\n",
        "            self.word2index[word] = self.n_words\n",
        "            self.word2count[word] = 1\n",
        "            self.index2word[self.n_words] = word\n",
        "            self.n_words += 1\n",
        "        else:\n",
        "            self.word2count[word] += 1\n",
        "    \n",
        "    def containsAllWords(self, sentence):\n",
        "        for word in sentence.split(' '):\n",
        "            if word not in self.word2index:\n",
        "              return False\n",
        "        return True"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "JxSCPH3lLFdA",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Preprocessing\n",
        "\n",
        "While loading data, it preprocesses the text data as follows:\n",
        "*   Normalize string by converting to ascii, lowercase, strip, and removing any non word characters.\n",
        "*   Filtering examples by word length and sentence length.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "iGAMN_f0mPed",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "# Turn a Unicode string to plain ASCII, thanks to\n",
        "# http://stackoverflow.com/a/518232/2809427\n",
        "def unicodeToAscii(s):\n",
        "    return ''.join(\n",
        "        c for c in unicodedata.normalize('NFD', s)\n",
        "        if unicodedata.category(c) != 'Mn'\n",
        "    )\n",
        "\n",
        "# Lowercase, trim, and remove non-letter characters\n",
        "def normalizeString(s):\n",
        "    s = unicodeToAscii(s.lower().strip())\n",
        "    s = re.sub(r\"[^a-z]+\", r\" \", s)\n",
        "    s = re.sub(r\"\\s+\", r\" \", s)\n",
        "    s.strip()\n",
        "    return s"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "qQHl9TlYjhUr",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Place this data set: https://drive.google.com/open?id=1Woa5zRbgpyNd3AA7ZMJXeSuvD4WaRaha into the corresponding path in Drive."
      ]
    },
    {
      "metadata": {
        "id": "eK0a_jUfmPef",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def readLangs():\n",
        "    print(\"Reading lines...\")\n",
        "\n",
        "    # Read the file and split into lines\n",
        "    lines = open(os.path.join(LOCAL_PATH, 'test.tsv'), encoding='utf-8').\\\n",
        "        read().strip().split('\\n')\n",
        "\n",
        "    # Split every line into pairs and normalize\n",
        "    examples = [[normalizeString(s) for s in l.split('\\t')[:3]] for l in lines]\n",
        "    input_lang = Lang('slang')\n",
        "    output_lang = Lang('explain')\n",
        "\n",
        "    return input_lang, output_lang, examples"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "33BYrFtDmPeh",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "MAX_SENTENCE_LENGTH = 25\n",
        "MAX_WORD_LENGTH = 10\n",
        "\n",
        "def filterExample(p):\n",
        "    return len(p) == 3 and \\\n",
        "        p[0].strip() and p[1].strip() and p[2].strip() and \\\n",
        "        len(p[2].split(' ')) < MAX_SENTENCE_LENGTH and \\\n",
        "        len(p[1].split(' ')) < MAX_SENTENCE_LENGTH and \\\n",
        "        len(p[0]) < MAX_WORD_LENGTH and \\\n",
        "        len(p[0].split(' ')) == 1 and \\\n",
        "        p[0] in p[2].split(' ')\n",
        "\n",
        "\n",
        "def filterExamples(examples):\n",
        "    return [example for example in examples if filterExample(example)]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "GxC7aIeDL2aO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Loading\n",
        "The following code should load data and show brief description of training examples."
      ]
    },
    {
      "metadata": {
        "id": "5ts9yrzfmPej",
        "colab_type": "code",
        "outputId": "7935c219-8528-42d2-9b2b-a9591f3e070d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 153
        }
      },
      "cell_type": "code",
      "source": [
        "MAX_WORDS_IN_LANG = 10000\n",
        "\n",
        "def prepareData():\n",
        "    input_lang, output_lang, examples = readLangs()\n",
        "    print(\"Read %s sentence examples\" % len(examples))\n",
        "    examples = filterExamples(examples)\n",
        "    print(\"Trimmed to %s sentence pairs\" % len(examples))\n",
        "    print(\"Counting words...\")\n",
        "    for example in examples:\n",
        "        input_lang.addSentence(example[2])\n",
        "        output_lang.addSentence(example[1])\n",
        "    print(\"Counted words:\")\n",
        "    print(input_lang.name, input_lang.n_words)\n",
        "    print(output_lang.name, output_lang.n_words)\n",
        "    return input_lang, output_lang, examples\n",
        "  \n",
        "input_lang, output_lang, examples = prepareData()\n",
        "print(random.choice(examples))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Reading lines...\n",
            "Read 75670 sentence examples\n",
            "Trimmed to 11045 sentence pairs\n",
            "Counting words...\n",
            "Counted words:\n",
            "slang 19525\n",
            "explain 16048\n",
            "['decker', 'part donkey part unicorn ', 'unicorn decker']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "8vtReyaMMFh5",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Splitting Training/Test examples"
      ]
    },
    {
      "metadata": {
        "id": "9JvJzlRtUPc5",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import random\n",
        "random.shuffle(examples)\n",
        "split_point = int(0.8 * len(examples))\n",
        "train_examples = examples[:split_point]\n",
        "test_examples = examples[split_point+1:]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "hj9gT0fuMOSn",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Saving Training/Test examples \n",
        "In order to reproduce the result, training/test examples are stored in Drive."
      ]
    },
    {
      "metadata": {
        "id": "lMlLqGOYPY96",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import pickle\n",
        "\n",
        "with open(os.path.join(LOCAL_PATH, 'saved_train_examples'), 'wb') as f:\n",
        "  pickle.dump((input_lang, output_lang, train_examples), f)\n",
        "with open(os.path.join(LOCAL_PATH, 'saved_test_examples'), 'wb') as f:\n",
        "  pickle.dump((input_lang, output_lang, test_examples), f)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "z8v-cYSkMjb3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Loading Training/Test examples "
      ]
    },
    {
      "metadata": {
        "id": "CpaTk_J4CBet",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import pickle\n",
        "\n",
        "with open(os.path.join(LOCAL_PATH, 'saved_train_examples'), 'rb') as f:\n",
        "  train_examples = pickle.load(f)[2]\n",
        "with open(os.path.join(LOCAL_PATH, 'saved_test_examples'), 'rb') as f:\n",
        "  obj = pickle.load(f)\n",
        "  test_examples = obj[2]\n",
        "  input_lang = obj[0]\n",
        "  output_lang = obj[1]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "_vCBgSkjmJUI",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Model Setup"
      ]
    },
    {
      "metadata": {
        "id": "WK3XYn_MMtaU",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## EncoderRNN\n",
        "Encoder to encode sequence of characters or words into embedding space along with hidden context."
      ]
    },
    {
      "metadata": {
        "id": "6JNIx5XgmPen",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class EncoderRNN(nn.Module):\n",
        "    def __init__(self, input_size, hidden_size):\n",
        "        super(EncoderRNN, self).__init__()\n",
        "        self.hidden_size = hidden_size\n",
        "\n",
        "        self.embedding = nn.Embedding(input_size, hidden_size)\n",
        "        self.gru = nn.GRU(hidden_size, hidden_size)\n",
        "\n",
        "    def forward(self, input, hidden):\n",
        "        embedded = self.embedding(input).view(1, 1, -1)\n",
        "        output = embedded\n",
        "        output, hidden = self.gru(output, hidden)\n",
        "        return output, hidden\n",
        "\n",
        "    def initHidden(self):\n",
        "        return torch.zeros(1, 1, self.hidden_size, device=device)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "VJ6h1W8BNDmm",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## TagEncoderRNN\n",
        "This encoder is meant for encoding sequence of word,tag pairs."
      ]
    },
    {
      "metadata": {
        "id": "WDH7L0ZzFdD4",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class TagEncoderRNN(nn.Module):\n",
        "    def __init__(self, input_size, tag_size, hidden_size):\n",
        "        super(TagEncoderRNN, self).__init__()\n",
        "        self.hidden_size = hidden_size\n",
        "        self.tag_size = tag_size\n",
        "        self.embedding = nn.Embedding(input_size, hidden_size)\n",
        "        self.gru = nn.GRU(hidden_size + tag_size, hidden_size)\n",
        "\n",
        "    def forward(self, input, tagTensor, hidden):\n",
        "        embedded = self.embedding(input).view(1, 1, -1)\n",
        "        tag_one_hot = torch.zeros(1, self.tag_size, device=device).scatter_(1, tagTensor.unsqueeze(1), 1.).view(1, 1, -1)\n",
        "        concat = torch.cat((embedded, tag_one_hot), 2)\n",
        "        output, hidden = self.gru(concat, hidden)\n",
        "        return output, hidden\n",
        "\n",
        "    def initHidden(self):\n",
        "        return torch.zeros(1, 1, self.hidden_size, device=device)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "UNFuFUc4NVEO",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## AttnDecoderRNN\n",
        "Decoder to decode from encoder output to generate sequence of explanantion words. In the beginning, it takes combined outputs from character level encoder and word level encoder and references all outpus of both encoders to leverage attention mechanism to predict the better sequences."
      ]
    },
    {
      "metadata": {
        "id": "HUpoxaUGd8wR",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class AttnDecoderRNN(nn.Module):\n",
        "    def __init__(self, hidden_size, output_size, dropout_p=0.05, \n",
        "                 context_max_length=MAX_SENTENCE_LENGTH,\n",
        "                 character_max_length=MAX_WORD_LENGTH):\n",
        "        super(AttnDecoderRNN, self).__init__()\n",
        "        self.hidden_size = hidden_size\n",
        "        self.output_size = output_size\n",
        "        self.dropout_p = dropout_p\n",
        "\n",
        "        self.embedding = nn.Embedding(self.output_size, self.hidden_size)\n",
        "        self.attn = nn.Linear(self.hidden_size * 2, context_max_length)\n",
        "        self.char_attn = nn.Linear(self.hidden_size * 2, character_max_length)\n",
        "        self.attn_combine = nn.Linear(self.hidden_size * 3, self.hidden_size)\n",
        "        self.dropout = nn.Dropout(self.dropout_p)\n",
        "        self.gru = nn.GRU(self.hidden_size, self.hidden_size)\n",
        "        self.out = nn.Linear(self.hidden_size, self.output_size)\n",
        "\n",
        "    def forward(self, input, hidden, encoder_outputs, char_encoder_outputs):\n",
        "        embedded = self.embedding(input).view(1, 1, -1)\n",
        "        embedded = self.dropout(embedded)\n",
        "\n",
        "        attn_weights = F.softmax(\n",
        "            self.attn(torch.cat((embedded[0], hidden[0]), 1)), dim=1)\n",
        "        attn_applied = torch.bmm(attn_weights.unsqueeze(0),\n",
        "                                 encoder_outputs.unsqueeze(0))\n",
        "        \n",
        "        char_attn_weights = F.softmax(\n",
        "            self.char_attn(torch.cat((embedded[0], hidden[0]), 1)), dim=1)\n",
        "        char_attn_applied = torch.bmm(char_attn_weights.unsqueeze(0),\n",
        "                                      char_encoder_outputs.unsqueeze(0))\n",
        "\n",
        "        output = torch.cat((embedded[0], attn_applied[0], char_attn_applied[0]), 1)\n",
        "        output = self.attn_combine(output).unsqueeze(0)\n",
        "\n",
        "        output = F.relu(output)\n",
        "        output, hidden = self.gru(output, hidden)\n",
        "\n",
        "        output = F.log_softmax(self.out(output[0]), dim=1)\n",
        "        return output, hidden, attn_weights\n",
        "\n",
        "    def initHidden(self):\n",
        "        return torch.zeros(1, 1, self.hidden_size, device=device)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "BiPDurvgOB4u",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Combiner\n",
        "Combiner to combine outputs of character level encoder and word level encoder as input of decoder."
      ]
    },
    {
      "metadata": {
        "id": "l0_xzrf9mPep",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "class Combiner(nn.Module):\n",
        "    def __init__(self, hidden_size):\n",
        "        super(Combiner, self).__init__()\n",
        "        self.hidden_size = hidden_size\n",
        "        self.linear = nn.Linear(hidden_size * 2, hidden_size)\n",
        "\n",
        "    def forward(self, word_hidden, char_hidden):\n",
        "        output = self.linear(torch.cat((word_hidden.view(-1), char_hidden.view(-1)), 0)).view(1, 1, -1)\n",
        "        return F.relu(output)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "aVH9hM__mOf6",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Model Training"
      ]
    },
    {
      "metadata": {
        "id": "2nNFzP_uONNH",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Setup"
      ]
    },
    {
      "metadata": {
        "id": "C4YKKHLQORcG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "The following functions will be used to create tensor variables to be used in training."
      ]
    },
    {
      "metadata": {
        "id": "O_mW4xdLmPeq",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def indexesFromSentence(lang, sentence):\n",
        "    sentence = sentence.strip()\n",
        "    return [lang.word2index[word] for word in sentence.split(' ')]\n",
        "  \n",
        "def tagIndexesFromSentence(lang, sentence):\n",
        "    sentence = sentence.strip()\n",
        "    return [lang.tag2index[tag] for word, tag in nltk.pos_tag(sentence.split(' '))]\n",
        "\n",
        "\n",
        "def tensorFromSentence(lang, sentence):\n",
        "    indexes = indexesFromSentence(lang, sentence)\n",
        "    indexes.append(EOS_token)\n",
        "    return torch.tensor(indexes, dtype=torch.long, device=device).view(-1, 1)\n",
        "  \n",
        "def tagTensorFromSentence(lang, sentence):\n",
        "    indexes = tagIndexesFromSentence(lang, sentence)\n",
        "    indexes.append(EOS_token)\n",
        "    return torch.tensor(indexes, dtype=torch.long, device=device).view(-1, 1)\n",
        "\n",
        "def charIndexesFromSentence(lang, sentence):\n",
        "    sentence = sentence.strip()\n",
        "    return [lang.char2index[char] for char in sentence]\n",
        "\n",
        "def charTensorFromSentence(lang, sentence):\n",
        "    indexes = charIndexesFromSentence(lang, sentence)\n",
        "    indexes.append(EOS_token)\n",
        "    return torch.tensor(indexes, dtype=torch.long, device=device).view(-1, 1) \n",
        "\n",
        "def tensorsFromExample(example):\n",
        "    input_tensor = tensorFromSentence(input_lang, example[2])\n",
        "    input_tag_tensor = tagTensorFromSentence(input_lang, example[2])\n",
        "    char_input_tensor = charTensorFromSentence(input_lang, example[0])\n",
        "    target_tensor = tensorFromSentence(output_lang, example[1])\n",
        "    target_tag_tensor = tagTensorFromSentence(output_lang, example[1])\n",
        "    return (input_tensor, char_input_tensor, target_tensor, input_tag_tensor)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "1Q8DigGdOaau",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Train()\n",
        "\n",
        "The function takes single training example, compute loss, and backprogate gradients. "
      ]
    },
    {
      "metadata": {
        "id": "xi8CPjNEmPey",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "teacher_forcing_ratio = 0.5\n",
        "\n",
        "def train(input_tensor, char_input_tensor, target_tensor, tag_tensor, encoder, char_encoder, decoder, combiner,\n",
        "          encoder_optimizer, char_encoder_optimizer, decoder_optimizer, combiner_optimizer, \n",
        "          criterion):\n",
        "    encoder_hidden = encoder.initHidden()\n",
        "    char_encoder_hidden = encoder.initHidden()\n",
        "\n",
        "    encoder_optimizer.zero_grad()\n",
        "    char_encoder_optimizer.zero_grad()\n",
        "    decoder_optimizer.zero_grad()\n",
        "    combiner_optimizer.zero_grad()\n",
        "\n",
        "    input_length = input_tensor.size(0)\n",
        "    char_input_length = char_input_tensor.size(0)\n",
        "    target_length = target_tensor.size(0)\n",
        "\n",
        "    loss = 0\n",
        "    \n",
        "    encoder_outputs = torch.zeros(MAX_SENTENCE_LENGTH, encoder.hidden_size, device=device)\n",
        "    char_encoder_outputs = torch.zeros(MAX_WORD_LENGTH, char_encoder.hidden_size, device=device)\n",
        "    \n",
        "    for ei in range(input_length):\n",
        "        encoder_output, encoder_hidden = encoder(\n",
        "            input_tensor[ei], tag_tensor[ei], encoder_hidden)\n",
        "        encoder_outputs[ei] = encoder_output[0, 0]\n",
        "\n",
        "    for ei in range(char_input_length):\n",
        "        char_encoder_output, char_encoder_hidden = char_encoder(\n",
        "            char_input_tensor[ei], char_encoder_hidden)\n",
        "        char_encoder_outputs[ei] = char_encoder_output[0, 0]\n",
        "\n",
        "    decoder_input = torch.tensor([[SOS_token]], device=device)\n",
        "    decoder_hidden = combiner(encoder_hidden, char_encoder_hidden)\n",
        "\n",
        "    use_teacher_forcing = True if random.random() < teacher_forcing_ratio else False\n",
        "\n",
        "    if use_teacher_forcing:\n",
        "        # Teacher forcing: Feed the target as the next input\n",
        "        for di in range(target_length):\n",
        "            decoder_output, decoder_hidden, decoder_attention = decoder(\n",
        "                decoder_input, decoder_hidden, encoder_outputs, char_encoder_outputs)\n",
        "            loss += criterion(decoder_output, target_tensor[di])\n",
        "            decoder_input = target_tensor[di]  # Teacher forcing\n",
        "\n",
        "    else:\n",
        "        # Without teacher forcing: use its own predictions as the next input\n",
        "        for di in range(target_length):\n",
        "            decoder_output, decoder_hidden, decoder_attention = decoder(\n",
        "                decoder_input, decoder_hidden, encoder_outputs, char_encoder_outputs)\n",
        "            topv, topi = decoder_output.topk(1)\n",
        "            decoder_input = topi.squeeze().detach()  # detach from history as input\n",
        "\n",
        "            loss += criterion(decoder_output, target_tensor[di])\n",
        "            if decoder_input.item() == EOS_token:\n",
        "                break\n",
        "\n",
        "    loss.backward()\n",
        "    encoder_optimizer.step()\n",
        "    char_encoder_optimizer.step()\n",
        "    decoder_optimizer.step()\n",
        "    combiner_optimizer.step()\n",
        "\n",
        "    return loss.item() / target_length"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "c0esrBwMPJ5j",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## writeModel()\n",
        "This function writes the model's current state into Drive.\n"
      ]
    },
    {
      "metadata": {
        "id": "Mnw8S5pzWJVE",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def writeModel(name, iter, encoder, char_encoder, decoder, combiner, encoder_optimizer, \n",
        "               char_encoder_optimizer, decoder_optimizer, combiner_optimizer, loss):\n",
        "  path = os.path.join(LOCAL_PATH, 'model-%s' % name)\n",
        "  torch.save({\n",
        "      'epoch': iter,\n",
        "      'encoder_state_dict': encoder.state_dict(),\n",
        "      'encoder_optimizer_state_dict': encoder_optimizer.state_dict(),\n",
        "      'char_encoder_state_dict': char_encoder.state_dict(),\n",
        "      'char_encoder_optimizer_state_dict': char_encoder_optimizer.state_dict(),\n",
        "      'decoder_state_dict': decoder.state_dict(),\n",
        "      'decoder_optimizer_state_dict': decoder_optimizer.state_dict(),\n",
        "      'combiner_state_dict': combiner.state_dict(),\n",
        "      'combiner_optimizer_state_dict': combiner_optimizer.state_dict(),\n",
        "      'loss': loss,\n",
        "      }, path)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "BwslXNodPWLT",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## loadModel()\n",
        "This function load the model from Drive"
      ]
    },
    {
      "metadata": {
        "id": "2T9gLn6DW9ZO",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def loadModel(name, encoder, char_encoder, decoder, combiner, encoder_optimizer, char_encoder_optimizer,\n",
        "         decoder_optimizer, combiner_optimizer):\n",
        "  path = os.path.join(LOCAL_PATH, 'model-%s' % name)\n",
        "  checkpoint = torch.load(path)\n",
        "  encoder.load_state_dict(checkpoint['encoder_state_dict'])\n",
        "  encoder_optimizer.load_state_dict(checkpoint['encoder_optimizer_state_dict'])\n",
        "  char_encoder.load_state_dict(checkpoint['char_encoder_state_dict'])\n",
        "  char_encoder_optimizer.load_state_dict(checkpoint['char_encoder_optimizer_state_dict'])\n",
        "  decoder.load_state_dict(checkpoint['decoder_state_dict'])\n",
        "  decoder_optimizer.load_state_dict(checkpoint['decoder_optimizer_state_dict'])\n",
        "  combiner.load_state_dict(checkpoint['combiner_state_dict'])\n",
        "  combiner_optimizer.load_state_dict(checkpoint['combiner_optimizer_state_dict'])\n",
        "  iter = checkpoint[\"epoch\"]\n",
        "  loss = checkpoint[\"loss\"]\n",
        "  return (iter, loss)\n",
        "  "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "iSqyf3NuPeeS",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## trainIters()\n",
        "\n",
        "The functions takes (word) encoder, char_encoder, decoder, combiner and train the them n iterations calling Train() function.\n",
        "\n",
        "The intermediate state of models being trained will be periodically stored in Drive every \"print_every\" iterations.\n",
        "\n",
        "It also supports loading the model and training n more iterations from the loaded state by setting load true and pass the model name.\n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "WT7-vCQ9mPe2",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "import time\n",
        "import math\n",
        "import matplotlib.pyplot as plt\n",
        "plt.switch_backend('agg')\n",
        "import matplotlib.ticker as ticker\n",
        "import numpy as np\n",
        "\n",
        "\n",
        "def showPlot(points):\n",
        "    plt.figure()\n",
        "    fig, ax = plt.subplots()\n",
        "    # this locator puts ticks at regular intervals\n",
        "    loc = ticker.MultipleLocator(base=0.2)\n",
        "    ax.yaxis.set_major_locator(loc)\n",
        "    plt.plot(points)\n",
        "\n",
        "def asMinutes(s):\n",
        "    m = math.floor(s / 60)\n",
        "    s -= m * 60\n",
        "    return '%dm %ds' % (m, s)\n",
        "\n",
        "\n",
        "def timeSince(since, percent):\n",
        "    now = time.time()\n",
        "    s = now - since\n",
        "    es = s / (percent)\n",
        "    rs = es - s\n",
        "    return '%s (- %s)' % (asMinutes(s), asMinutes(rs))\n",
        "  \n",
        "def trainIters(encoder, char_encoder, decoder, combiner, n_iters, print_every=1000, plot_every=100, learning_rate=0.0001, \n",
        "               load=False, name=None):\n",
        "    start = time.time()\n",
        "    plot_losses = []\n",
        "    print_loss_total = 0  # Reset every print_every\n",
        "    plot_loss_total = 0  # Reset every plot_every\n",
        "\n",
        "    encoder_optimizer = optim.Adam(encoder.parameters(), lr=learning_rate)\n",
        "    char_encoder_optimizer = optim.Adam(char_encoder.parameters(), lr=learning_rate)\n",
        "    decoder_optimizer = optim.Adam(decoder.parameters(), lr=learning_rate)\n",
        "    combiner_optimizer = optim.Adam(combiner.parameters(), lr=learning_rate)\n",
        "    criterion = nn.NLLLoss()\n",
        "    \n",
        "    start_iter = 0\n",
        "    start_loss = 0\n",
        "    \n",
        "    if load:\n",
        "      start_iter, start_loss = loadModel(name, encoder, char_encoder, decoder, combiner, \n",
        "                                         encoder_optimizer, char_encoder_optimizer,\n",
        "                                         decoder_optimizer, combiner_optimizer)\n",
        "      print('loaded model: %s start_iter: %d loss: %s' % (name, start_iter, start_loss))\n",
        "\n",
        "    for iter in range(1, n_iters + 1):\n",
        "        training_pair = tensorsFromExample(train_examples[(iter - 1) % len(train_examples)])\n",
        "        input_tensor = training_pair[0]\n",
        "        char_input_tensor = training_pair[1]\n",
        "        target_tensor = training_pair[2]\n",
        "        tag_tensor = training_pair[3]\n",
        "\n",
        "        loss = train(input_tensor, char_input_tensor, target_tensor, tag_tensor, encoder, char_encoder, decoder, combiner,\n",
        "                     encoder_optimizer, char_encoder_optimizer, decoder_optimizer, combiner_optimizer, criterion)\n",
        "        print_loss_total += loss\n",
        "        plot_loss_total += loss\n",
        "\n",
        "        if iter % print_every == 0:\n",
        "            print_loss_avg = print_loss_total / print_every\n",
        "            print_loss_total = 0\n",
        "            print('%s (%d %d%%) %.4f' % (timeSince(start, iter / n_iters),\n",
        "                                         iter + start_iter, iter / n_iters * 100, print_loss_avg))\n",
        "            writeModel(name, iter + start_iter, encoder, char_encoder, decoder, combiner, encoder_optimizer, \n",
        "               char_encoder_optimizer, decoder_optimizer, combiner_optimizer, print_loss_avg)\n",
        "\n",
        "        if iter % plot_every == 0:\n",
        "            plot_loss_avg = plot_loss_total / plot_every\n",
        "            plot_losses.append(plot_loss_avg)\n",
        "            plot_loss_total = 0\n",
        "    showPlot(plot_losses)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "TrU0-jpiTDfj",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Train models\n",
        "\n",
        "Currently, it trains the model with 1000 iterations for demo but should be much bigger to train good quality model."
      ]
    },
    {
      "metadata": {
        "id": "LM56WgUjmPe7",
        "colab_type": "code",
        "outputId": "b3d1f086-8427-4e05-874c-9dbd038c2904",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        }
      },
      "cell_type": "code",
      "source": [
        "hidden_size = 512\n",
        "encoder1 = TagEncoderRNN(input_lang.n_words, input_lang.n_tags, hidden_size).to(device)\n",
        "encoder2 = EncoderRNN(input_lang.n_chars, hidden_size).to(device)\n",
        "attn_decoder1 = AttnDecoderRNN(hidden_size, output_lang.n_words).to(device)\n",
        "combiner = Combiner(hidden_size).to(device)\n",
        "trainIters(encoder1, encoder2, attn_decoder1, combiner, 1000, print_every=100, load=False, name=\"medium_tag_rerun2\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0m 11s (- 1m 43s) (100 10%) 6.6759\n",
            "0m 23s (- 1m 35s) (200 20%) 4.9694\n",
            "0m 36s (- 1m 25s) (300 30%) 4.9850\n",
            "0m 49s (- 1m 13s) (400 40%) 5.2649\n",
            "1m 2s (- 1m 2s) (500 50%) 5.3848\n",
            "1m 15s (- 0m 50s) (600 60%) 5.1692\n",
            "1m 30s (- 0m 38s) (700 70%) 6.1309\n",
            "1m 44s (- 0m 26s) (800 80%) 5.7058\n",
            "1m 58s (- 0m 13s) (900 90%) 5.6033\n",
            "2m 11s (- 0m 0s) (1000 100%) 5.5083\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "RKokn8jcmSnl",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Evaluation"
      ]
    },
    {
      "metadata": {
        "id": "g8Zry01jTRRL",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Setup evaluation"
      ]
    },
    {
      "metadata": {
        "id": "3pdWopFrmPe-",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def evaluate(encoder, char_encoder, decoder, combiner, sentence, target_words,\n",
        "             context_max_length=MAX_SENTENCE_LENGTH, character_max_length=MAX_WORD_LENGTH):\n",
        "    with torch.no_grad():\n",
        "        encoder_hidden = encoder.initHidden()\n",
        "        char_encoder_hidden = encoder.initHidden()\n",
        "        input_tensor = tensorFromSentence(input_lang, sentence)\n",
        "        input_tag_tensor = tagTensorFromSentence(input_lang, sentence)\n",
        "        input_length = input_tensor.size()[0]\n",
        "        char_input_tensor = charTensorFromSentence(input_lang, target_words)\n",
        "        char_input_length = char_input_tensor.size()[0]\n",
        "        \n",
        "        encoder_hidden = encoder.initHidden()\n",
        "        encoder_outputs = torch.zeros(context_max_length, encoder.hidden_size, device=device)\n",
        "        for ei in range(input_length):\n",
        "            encoder_output, encoder_hidden = encoder(input_tensor[ei], input_tag_tensor[ei],\n",
        "                                                     encoder_hidden)\n",
        "            encoder_outputs[ei] += encoder_output[0, 0]\n",
        "        \n",
        "        char_encoder_hidden = char_encoder.initHidden()\n",
        "        char_encoder_outputs = torch.zeros(character_max_length, char_encoder.hidden_size, device=device)\n",
        "        for ei in range(char_input_length):\n",
        "            char_encoder_output, char_encoder_hidden = char_encoder(char_input_tensor[ei],\n",
        "                                                                    char_encoder_hidden)\n",
        "            char_encoder_outputs[ei] += char_encoder_output[0, 0]\n",
        "\n",
        "        decoder_input = torch.tensor([[SOS_token]], device=device)  # SOS\n",
        "        decoder_hidden = combiner(encoder_hidden, char_encoder_hidden)\n",
        "\n",
        "        decoded_words = []\n",
        "\n",
        "        for di in range(context_max_length):\n",
        "            decoder_output, decoder_hidden, decoder_attention = decoder(\n",
        "                decoder_input, decoder_hidden, encoder_outputs, char_encoder_outputs)\n",
        "            topv, topi = decoder_output.data.topk(1)\n",
        "            if topi.item() == EOS_token:\n",
        "                decoded_words.append('<EOS>')\n",
        "                break\n",
        "            else:\n",
        "                decoded_words.append(output_lang.index2word[topi.item()])\n",
        "\n",
        "            decoder_input = topi.squeeze().detach()\n",
        "\n",
        "        return decoded_words"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "e1husE3NmPfA",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def evaluateRandomly(encoder, char_encoder, decoder, combiner, n=10):\n",
        "    for i in range(n):\n",
        "        example = test_examples[i]\n",
        "        print('>', example[0])\n",
        "        print('=', example[1])\n",
        "        output_words = evaluate(encoder, char_encoder, decoder, combiner, example[2], example[0])\n",
        "        output_sentence = ' '.join(output_words)\n",
        "        print('<', output_sentence)\n",
        "        print('')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "iktkHeitTTzW",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## Qualtitative Evaluation\n",
        "\n",
        "Randomly sampling examples and compare predicted explanation and actual explanation of the examples."
      ]
    },
    {
      "metadata": {
        "id": "wrbhfwJPmPfE",
        "colab_type": "code",
        "outputId": "f417d0ca-69ba-4f20-f1d5-07a76a04a360",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 717
        }
      },
      "cell_type": "code",
      "source": [
        "evaluateRandomly(encoder1, encoder2, attn_decoder1, combiner)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "> uscg\n",
            "= united states coast guard \n",
            "< the of a of a <EOS>\n",
            "\n",
            "> slunky\n",
            "= a combination of a monkey and a slut therefore creating a slunky \n",
            "< a who of a <EOS>\n",
            "\n",
            "> pelham\n",
            "= a neat town in deep southern westchester you can actually walk to pelham from the train like i did today cool place \n",
            "< the of a of a <EOS>\n",
            "\n",
            "> droideka\n",
            "= a little snot from orange county with a big nose and no tits \n",
            "< a who of a <EOS>\n",
            "\n",
            "> chezed\n",
            "= when your mind can t think straight usually from being extremely high originated from ge crew\n",
            "< the of a of <EOS>\n",
            "\n",
            "> chelfie\n",
            "=  chinese selfie to take a selfie with nothing significant around or for no particular reason \n",
            "< a who of a <EOS>\n",
            "\n",
            "> brej\n",
            "= abbreviation of bredjrin a close friend \n",
            "< the of a of <EOS>\n",
            "\n",
            "> porgie\n",
            "= a weak heart person who is afraid to fight or has fear of being beaten in a fight \n",
            "< a who of a <EOS>\n",
            "\n",
            "> babam\n",
            "= a word of excitement is the response to everything\n",
            "< the of a of <EOS>\n",
            "\n",
            "> burbsies\n",
            "= heavy\n",
            "< the of a of <EOS>\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "cNajyIePTmnX",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "## BLEU\n",
        "\n",
        "BLEU score will be used to quantitatively evaluate the model."
      ]
    },
    {
      "metadata": {
        "id": "3fpwoQQmu1yc",
        "colab_type": "code",
        "outputId": "0ac09072-f882-4106-896a-c563900f0e24",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "import nltk\n",
        "\n",
        "entries = []\n",
        "count = 0.0\n",
        "total_score = 0.0\n",
        "for example in test_examples: \n",
        "  reference = example[1]\n",
        "  target = example[0]\n",
        "  explain = evaluate(encoder1, encoder2, attn_decoder1, combiner, example[2], example[0])\n",
        "  explain = [x for x in explain[:-1] if x != '']\n",
        "  score = nltk.translate.bleu_score.sentence_bleu([reference.split()], explain, weights=[1.0])\n",
        "  total_score += score\n",
        "  count += 1.0\n",
        "  entries.append((score, (target, ' '.join(explain), reference)))\n",
        "print(total_score / count)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0.061706882237824355\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "TRa6-UQW9cn9",
        "colab_type": "code",
        "outputId": "a5dd6b11-1de0-4dd0-ed9b-54d1a198d101",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 187
        }
      },
      "cell_type": "code",
      "source": [
        "sorted(entries, reverse=True)[:10]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[(0.6, ('raydz', 'the of a of a', 'a protector of a group ')),\n",
              " (0.6, ('drullet', 'the of a of a', 'the mullet of a dragon ')),\n",
              " (0.5, ('yolology', 'the of a of', 'the study of yolo')),\n",
              " (0.5, ('shuffle', 'the of a of', ' the mixing of cards ')),\n",
              " (0.5, ('paregoric', 'the of a of', 'a tincture of opium')),\n",
              " (0.5, ('keswick', 'the of a of', 'the armpit of ontario ')),\n",
              " (0.5, ('jbone', 'a who of a', 'a joint of marijuana')),\n",
              " (0.5, ('gebs', 'the of a of', 'the feeling of pooing ')),\n",
              " (0.5, ('encrypt', 'the of a of', 'the act of encrypting ')),\n",
              " (0.5, ('crapatola', 'the of a of', 'a lot of crap'))]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 48
        }
      ]
    },
    {
      "metadata": {
        "id": "kV8tB76kZ4T3",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "# Trained Model"
      ]
    },
    {
      "metadata": {
        "id": "v5Yrg71diXlK",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "*   training_example: https://drive.google.com/open?id=15M1JtB8IyaT-fedyeKRPBlF71z242Az7\n",
        "*   test_example: https://drive.google.com/open?id=1liTQoeaxVqxyAGWnmDYfrnwU7QiuNK3U\n",
        "\n",
        "These training/test examples are the same dataset used in training the following models. Place them in Drive and load them to evaluate in the same test set.\n",
        "\n"
      ]
    },
    {
      "metadata": {
        "id": "DS8h5RhPdrDx",
        "colab_type": "code",
        "outputId": "f803ac5a-c917-42cb-ed4b-1a18570f7de3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        }
      },
      "cell_type": "code",
      "source": [
        "import pickle\n",
        "\n",
        "input_lang, output_lang, examples = prepareData()\n",
        "with open(os.path.join(LOCAL_PATH, 'training/train_examples'), 'rb') as f:\n",
        "  train_examples = pickle.load(f)[2]\n",
        "with open(os.path.join(LOCAL_PATH, 'training/test_examples'), 'rb') as f:\n",
        "  obj = pickle.load(f)\n",
        "  test_examples = obj[2]"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Reading lines...\n",
            "Read 75670 sentence examples\n",
            "Trimmed to 11045 sentence pairs\n",
            "Counting words...\n",
            "Counted words:\n",
            "slang 19525\n",
            "explain 16048\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "W4gKT9uDgxnz",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Place trained models using POS tag in Drive data path. \n",
        "trained model: https://drive.google.com/open?id=19PrgCX6-8ocGD_l4ltVVU-6YsuWz9vwA"
      ]
    },
    {
      "metadata": {
        "id": "XxgvImOSZ7ka",
        "colab_type": "code",
        "outputId": "d3c63a9d-dc9b-4f37-cae0-e4b853a5f8ee",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "hidden_size = 512\n",
        "trained_tag_encoder = TagEncoderRNN(input_lang.n_words, input_lang.n_tags, hidden_size).to(device)\n",
        "trained_tag_character_encoder = EncoderRNN(input_lang.n_chars, hidden_size).to(device)\n",
        "trained_tag_attn_decoder = AttnDecoderRNN(hidden_size, output_lang.n_words).to(device)\n",
        "trained_tag_combiner = Combiner(hidden_size).to(device)\n",
        "trainIters(trained_tag_encoder, trained_tag_character_encoder, \n",
        "           trained_tag_attn_decoder, trained_tag_combiner, 0, \n",
        "           print_every=100, load=True, name=\"trained_tag_model\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "loaded model: trained_tag_model start_iter: 301000 loss: 0.6147021469266243\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "JWpJ1SqqakNV",
        "colab_type": "code",
        "outputId": "dd4600bc-abb5-436f-cf24-197c20741ed7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "entries = []\n",
        "count = 0.0\n",
        "total_score = 0.0\n",
        "for example in test_examples: \n",
        "  reference = example[1]\n",
        "  target = example[0]\n",
        "  explain = evaluate(trained_tag_encoder, trained_tag_character_encoder, \n",
        "                     trained_tag_attn_decoder, trained_tag_combiner, example[2], example[0])\n",
        "  explain = [x for x in explain[:-1] if x != '']\n",
        "  score = nltk.translate.bleu_score.sentence_bleu([reference.split()], explain, weights=[1.0])\n",
        "  total_score += score\n",
        "  count += 1.0\n",
        "  entries.append((score, (target, ' '.join(explain), reference)))\n",
        "print(total_score / count)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0.6554927882965749\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "f5Pcky6zhAAG",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "Place trained models without POS tag in Drive. Trained model: https://drive.google.com/open?id=1cJY-4KDsh04EBEh6HeX8G4o75sKcL5Bt"
      ]
    },
    {
      "metadata": {
        "id": "YmLhD2wler-t",
        "colab_type": "code",
        "outputId": "e48c9eac-c495-47f2-ccb2-2c8659398288",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "hidden_size = 512\n",
        "trained_nontag_encoder = EncoderRNN(input_lang.n_words, hidden_size).to(device)\n",
        "trained_nontag_character_encoder = EncoderRNN(input_lang.n_chars, hidden_size).to(device)\n",
        "trained_nontag_attn_decoder = AttnDecoderRNN(hidden_size, output_lang.n_words).to(device)\n",
        "trained_nontag_combiner = Combiner(hidden_size).to(device)\n",
        "trainIters(trained_nontag_encoder, trained_nontag_character_encoder, \n",
        "           trained_nontag_attn_decoder, trained_nontag_combiner, 0, \n",
        "           print_every=100, load=True, name=\"medium-rerun\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "loaded model: medium-rerun start_iter: 300000 loss: 0.4249174690253783\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "MU9Xa0ZOfDU0",
        "colab_type": "code",
        "colab": {}
      },
      "cell_type": "code",
      "source": [
        "def evaluate_nontag(encoder, char_encoder, decoder, combiner, sentence, target_words,\n",
        "             context_max_length=MAX_SENTENCE_LENGTH, character_max_length=MAX_WORD_LENGTH):\n",
        "    with torch.no_grad():\n",
        "        encoder_hidden = encoder.initHidden()\n",
        "        char_encoder_hidden = encoder.initHidden()\n",
        "        input_tensor = tensorFromSentence(input_lang, sentence)\n",
        "        input_length = input_tensor.size()[0]\n",
        "        char_input_tensor = charTensorFromSentence(input_lang, target_words)\n",
        "        char_input_length = char_input_tensor.size()[0]\n",
        "        \n",
        "        encoder_hidden = encoder.initHidden()\n",
        "        encoder_outputs = torch.zeros(context_max_length, encoder.hidden_size, device=device)\n",
        "        for ei in range(input_length):\n",
        "            encoder_output, encoder_hidden = encoder(input_tensor[ei],\n",
        "                                                     encoder_hidden)\n",
        "            encoder_outputs[ei] += encoder_output[0, 0]\n",
        "        \n",
        "        char_encoder_hidden = char_encoder.initHidden()\n",
        "        char_encoder_outputs = torch.zeros(character_max_length, char_encoder.hidden_size, device=device)\n",
        "        for ei in range(char_input_length):\n",
        "            char_encoder_output, char_encoder_hidden = char_encoder(char_input_tensor[ei],\n",
        "                                                                    char_encoder_hidden)\n",
        "            char_encoder_outputs[ei] += char_encoder_output[0, 0]\n",
        "\n",
        "        decoder_input = torch.tensor([[SOS_token]], device=device)  # SOS\n",
        "        decoder_hidden = combiner(encoder_hidden, char_encoder_hidden)\n",
        "\n",
        "        decoded_words = []\n",
        "\n",
        "        for di in range(context_max_length):\n",
        "            decoder_output, decoder_hidden, decoder_attention = decoder(\n",
        "                decoder_input, decoder_hidden, encoder_outputs, char_encoder_outputs)\n",
        "            topv, topi = decoder_output.data.topk(1)\n",
        "            if topi.item() == EOS_token:\n",
        "                decoded_words.append('<EOS>')\n",
        "                break\n",
        "            else:\n",
        "                decoded_words.append(output_lang.index2word[topi.item()])\n",
        "\n",
        "            decoder_input = topi.squeeze().detach()\n",
        "\n",
        "        return decoded_words"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "metadata": {
        "id": "aa5fJ0lxfPIq",
        "colab_type": "code",
        "outputId": "3adc50c1-0a6c-48f8-e3c7-558ca7c12ac7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "cell_type": "code",
      "source": [
        "entries = []\n",
        "count = 0.0\n",
        "total_score = 0.0\n",
        "for example in test_examples: \n",
        "  reference = example[1]\n",
        "  target = example[0]\n",
        "  explain = evaluate_nontag(trained_nontag_encoder, trained_nontag_character_encoder, \n",
        "                     trained_nontag_attn_decoder, trained_nontag_combiner, example[2], example[0])\n",
        "  explain = [x for x in explain[:-1] if x != '']\n",
        "  score = nltk.translate.bleu_score.sentence_bleu([reference.split()], explain, weights=[1.0])\n",
        "  total_score += score\n",
        "  count += 1.0\n",
        "  entries.append((score, (target, ' '.join(explain), reference)))\n",
        "print(total_score / count)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "0.4350372324751226\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "metadata": {
        "id": "FjRJTeSfhYPa",
        "colab_type": "text"
      },
      "cell_type": "markdown",
      "source": [
        "As evaluation of the two models show, POS tag feature helps the quality of the prediction."
      ]
    }
  ]
}