NLP/Notebooks/10_Seq2Seq_con_Transformers.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "kernelspec": {
      "display_name": "Python 3",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.12"
    },
    "colab": {
      "name": "10_Seq2Seq con Transformers.ipynb",
      "provenance": [],
      "collapsed_sections": []
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Pmv31OyGW0I8"
      },
      "source": [
        "\n",
        "Traducción de idiomas con nn.Transformer y torchtext\n",
        "======================================================\n",
        "\n",
        "Este tutorial muestra cómo entrenar un modelo de traducción desde cero usando Transformer. Usaremos el conjunto de datos [Multi30k](http://www.statmt.org/wmt16/multimodal-task.html#task1) para entrenar un modelo de traducción del alemán al inglés.\n",
        "\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5qGquiL7W0JY"
      },
      "source": [
        "Revisar\n",
        "----------\n",
        "\n",
        "1. [Attention is all you need paper](https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf)\n",
        "2. [The annotated transformer](https://nlp.seas.harvard.edu/2018/04/03/attention.html#positional-encoding)\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JdpOrYjqW9ll",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "0b02af5f-c3e0-4223-8424-c3a041caa50b"
      },
      "source": [
        "!pip install -U spacy > /dev/null\n",
        "!pip install torchtext > /dev/null\n",
        "!python -m spacy download en_core_web_sm > /dev/null\n",
        "!python -m spacy download de_core_news_sm > /dev/null"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "2021-07-15 00:44:27.251242: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0\n",
            "2021-07-15 00:44:34.891014: I tensorflow/stream_executor/platform/default/dso_loader.cc:53] Successfully opened dynamic library libcudart.so.11.0\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aaC9x21sW0I6"
      },
      "source": [
        "%matplotlib inline"
      ],
      "execution_count": 6,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kDg0G61yW0I9"
      },
      "source": [
        "Procesamiento y abastecimiento de datos\n",
        "----------------------------\n",
        "\n",
        "La librería de [torchtext](https://pytorch.org/text/stable/) tiene utilidades para crear conjuntos de datos que se pueden iterar fácilmente con el fin de crear un modelo de traducción de idiomas. En este ejemplo, mostramos cómo usar los conjuntos de datos incorporados de torchtext, tokenizar una oración de texto sin formato, construir vocabulario y numerar tokens en tensor. Usaremos\n",
        "el dataset [Multi30k](https://pytorch.org/text/stable/datasets.html#multi30k) que produce un par de oraciones sin formato source-target. \n",
        "\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uw0OfcioW0I-"
      },
      "source": [
        "from torchtext.data.utils import get_tokenizer\n",
        "from torchtext.vocab import build_vocab_from_iterator\n",
        "from torchtext.datasets import Multi30k\n",
        "from typing import Iterable, List\n",
        "\n",
        "\n",
        "SRC_LANGUAGE = 'de'\n",
        "TGT_LANGUAGE = 'en'\n",
        "\n",
        "# Place-holders\n",
        "token_transform = {}\n",
        "vocab_transform = {}\n",
        "\n",
        "\n",
        "# Cree un tokenizador de idioma de origen y de destino. Asegúrate de instalar las dependencias\n",
        "# pip install -U spacy\n",
        "# python -m spacy download en_core_web_sm\n",
        "# python -m spacy download de_core_news_sm\n",
        "token_transform[SRC_LANGUAGE] = get_tokenizer('spacy', language='de_core_news_sm')\n",
        "token_transform[TGT_LANGUAGE] = get_tokenizer('spacy', language='en_core_web_sm')\n",
        "\n",
        "\n",
        "# función auxiliar para producir una lista de tokens\n",
        "def yield_tokens(data_iter: Iterable, language: str) -> List[str]:\n",
        "    language_index = {SRC_LANGUAGE: 0, TGT_LANGUAGE: 1}\n",
        "\n",
        "    for data_sample in data_iter:\n",
        "        yield token_transform[language](data_sample[language_index[language]])\n",
        "\n",
        "# Definir símbolos e índices especiales\n",
        "UNK_IDX, PAD_IDX, BOS_IDX, EOS_IDX = 0, 1, 2, 3\n",
        "# Make sure the tokens are in order of their indices to properly insert them in vocab\n",
        "special_symbols = ['<unk>', '<pad>', '<bos>', '<eos>']\n",
        " \n",
        "for ln in [SRC_LANGUAGE, TGT_LANGUAGE]:\n",
        "    # Training data Iterator \n",
        "    train_iter = Multi30k(split='train', language_pair=(SRC_LANGUAGE, TGT_LANGUAGE))\n",
        "    # Create torchtext's Vocab object \n",
        "    vocab_transform[ln] = build_vocab_from_iterator(yield_tokens(train_iter, ln),\n",
        "                                                    min_freq=1,\n",
        "                                                    specials=special_symbols,\n",
        "                                                    special_first=True)\n",
        "\n",
        "# Set UNK_IDX as the default index. This index is returned when the token is not found. \n",
        "# If not set, it throws RuntimeError when the queried token is not found in the Vocabulary. \n",
        "for ln in [SRC_LANGUAGE, TGT_LANGUAGE]:\n",
        "  vocab_transform[ln].set_default_index(UNK_IDX)"
      ],
      "execution_count": 7,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Q7oH6GQrW0JA"
      },
      "source": [
        "Red Seq2Seq usando Transformer\n",
        "---------------------------------\n",
        "\n",
        "Transformer es un modelo introducido en el paper [\"Attention is all you need\"](https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf) para resolver tareas de traducción automática. A continuación, crearemos una red Seq2Seq que utiliza Transformer. La red consta de tres partes. La primera parte es la capa de incrustación. Esta capa convierte el tensor de los índices de entrada en el tensor correspondiente de las incrustaciones de entrada. Estas incrustaciones se aumentan aún más con codificaciones posicionales para proporcionar información de posición de los tokens de entrada al modelo. La segunda parte es el modelo [Transformer](https://pytorch.org/docs/stable/generated/torch.nn.Transformer.html) real. Finalmente, la salida del modelo Transformer se pasa a través de una capa lineal que proporciona probabilidades no normalizadas para cada token en el idioma de destino.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GFtiovSKW0JC"
      },
      "source": [
        "from torch import Tensor\n",
        "import torch\n",
        "import torch.nn as nn\n",
        "from torch.nn import Transformer\n",
        "import math\n",
        "DEVICE = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n",
        "\n",
        "# Módulo auxiliar que agrega codificación posicional a la incrustación del token \n",
        "# para introducir una noción de orden de palabras\n",
        "class PositionalEncoding(nn.Module):\n",
        "    def __init__(self,\n",
        "                 emb_size: int,\n",
        "                 dropout: float,\n",
        "                 maxlen: int = 5000):\n",
        "        super(PositionalEncoding, self).__init__()\n",
        "        den = torch.exp(- torch.arange(0, emb_size, 2)* math.log(10000) / emb_size)\n",
        "        pos = torch.arange(0, maxlen).reshape(maxlen, 1)\n",
        "        pos_embedding = torch.zeros((maxlen, emb_size))\n",
        "        pos_embedding[:, 0::2] = torch.sin(pos * den)\n",
        "        pos_embedding[:, 1::2] = torch.cos(pos * den)\n",
        "        pos_embedding = pos_embedding.unsqueeze(-2)\n",
        "\n",
        "        self.dropout = nn.Dropout(dropout)\n",
        "        self.register_buffer('pos_embedding', pos_embedding)\n",
        "\n",
        "    def forward(self, token_embedding: Tensor):\n",
        "        return self.dropout(token_embedding + self.pos_embedding[:token_embedding.size(0), :])\n",
        "\n",
        "# Módulo auxiliar para convertir el tensor de índices de entrada en el tensor \n",
        "# correspondiente de incrustaciones de tokens\n",
        "class TokenEmbedding(nn.Module):\n",
        "    def __init__(self, vocab_size: int, emb_size):\n",
        "        super(TokenEmbedding, self).__init__()\n",
        "        self.embedding = nn.Embedding(vocab_size, emb_size)\n",
        "        self.emb_size = emb_size\n",
        "\n",
        "    def forward(self, tokens: Tensor):\n",
        "        return self.embedding(tokens.long()) * math.sqrt(self.emb_size)\n",
        "\n",
        "# Red Seq2Seq\n",
        "class Seq2SeqTransformer(nn.Module):\n",
        "    def __init__(self,\n",
        "                 num_encoder_layers: int,\n",
        "                 num_decoder_layers: int,\n",
        "                 emb_size: int,\n",
        "                 nhead: int,\n",
        "                 src_vocab_size: int,\n",
        "                 tgt_vocab_size: int,\n",
        "                 dim_feedforward: int = 512,\n",
        "                 dropout: float = 0.1):\n",
        "        super(Seq2SeqTransformer, self).__init__()\n",
        "        self.transformer = Transformer(d_model=emb_size,\n",
        "                                       nhead=nhead,\n",
        "                                       num_encoder_layers=num_encoder_layers,\n",
        "                                       num_decoder_layers=num_decoder_layers,\n",
        "                                       dim_feedforward=dim_feedforward,\n",
        "                                       dropout=dropout)\n",
        "        self.generator = nn.Linear(emb_size, tgt_vocab_size)\n",
        "        self.src_tok_emb = TokenEmbedding(src_vocab_size, emb_size)\n",
        "        self.tgt_tok_emb = TokenEmbedding(tgt_vocab_size, emb_size)\n",
        "        self.positional_encoding = PositionalEncoding(\n",
        "            emb_size, dropout=dropout)\n",
        "\n",
        "    def forward(self,\n",
        "                src: Tensor,\n",
        "                trg: Tensor,\n",
        "                src_mask: Tensor,\n",
        "                tgt_mask: Tensor,\n",
        "                src_padding_mask: Tensor,\n",
        "                tgt_padding_mask: Tensor,\n",
        "                memory_key_padding_mask: Tensor):\n",
        "        src_emb = self.positional_encoding(self.src_tok_emb(src))\n",
        "        tgt_emb = self.positional_encoding(self.tgt_tok_emb(trg))\n",
        "        outs = self.transformer(src_emb, tgt_emb, src_mask, tgt_mask, None, \n",
        "                                src_padding_mask, tgt_padding_mask, memory_key_padding_mask)\n",
        "        return self.generator(outs)\n",
        "\n",
        "    def encode(self, src: Tensor, src_mask: Tensor):\n",
        "        return self.transformer.encoder(self.positional_encoding(\n",
        "                            self.src_tok_emb(src)), src_mask)\n",
        "\n",
        "    def decode(self, tgt: Tensor, memory: Tensor, tgt_mask: Tensor):\n",
        "        return self.transformer.decoder(self.positional_encoding(\n",
        "                          self.tgt_tok_emb(tgt)), memory,\n",
        "                          tgt_mask)"
      ],
      "execution_count": 8,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8hSjcvOkW0JJ"
      },
      "source": [
        "Durante el entrenamiento, necesitamos una máscara de palabras posterior que evitará que el modelo mire las palabras futuras al hacer predicciones. También necesitaremos máscaras para ocultar los tokens de relleno de origen y destino. A continuación, definamos una función que se encargará de ambos. \n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XmC3CjcsW0JO"
      },
      "source": [
        "def generate_square_subsequent_mask(sz):\n",
        "    mask = (torch.triu(torch.ones((sz, sz), device=DEVICE)) == 1).transpose(0, 1)\n",
        "    mask = mask.float().masked_fill(mask == 0, float('-inf')).masked_fill(mask == 1, float(0.0))\n",
        "    return mask\n",
        "\n",
        "\n",
        "def create_mask(src, tgt):\n",
        "    src_seq_len = src.shape[0]\n",
        "    tgt_seq_len = tgt.shape[0]\n",
        "\n",
        "    tgt_mask = generate_square_subsequent_mask(tgt_seq_len)\n",
        "    src_mask = torch.zeros((src_seq_len, src_seq_len),device=DEVICE).type(torch.bool)\n",
        "\n",
        "    src_padding_mask = (src == PAD_IDX).transpose(0, 1)\n",
        "    tgt_padding_mask = (tgt == PAD_IDX).transpose(0, 1)\n",
        "    return src_mask, tgt_mask, src_padding_mask, tgt_padding_mask"
      ],
      "execution_count": 9,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ElzqIVzPW0JP"
      },
      "source": [
        "Definamos ahora los parámetros de nuestro modelo y creemos una instancia de los mismos. A continuación, también definimos nuestra función de pérdida, que es la pérdida de entropía cruzada y el optimizador utilizado para el entrenamiento.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VJ82_YtSW0JQ"
      },
      "source": [
        "torch.manual_seed(0)\n",
        "\n",
        "SRC_VOCAB_SIZE = len(vocab_transform[SRC_LANGUAGE])\n",
        "TGT_VOCAB_SIZE = len(vocab_transform[TGT_LANGUAGE])\n",
        "EMB_SIZE = 512\n",
        "NHEAD = 8\n",
        "FFN_HID_DIM = 512\n",
        "BATCH_SIZE = 128\n",
        "NUM_ENCODER_LAYERS = 3\n",
        "NUM_DECODER_LAYERS = 3\n",
        "\n",
        "transformer = Seq2SeqTransformer(NUM_ENCODER_LAYERS, NUM_DECODER_LAYERS, EMB_SIZE, \n",
        "                                 NHEAD, SRC_VOCAB_SIZE, TGT_VOCAB_SIZE, FFN_HID_DIM)\n",
        "\n",
        "for p in transformer.parameters():\n",
        "    if p.dim() > 1:\n",
        "        nn.init.xavier_uniform_(p)\n",
        "\n",
        "transformer = transformer.to(DEVICE)\n",
        "\n",
        "loss_fn = torch.nn.CrossEntropyLoss(ignore_index=PAD_IDX)\n",
        "\n",
        "optimizer = torch.optim.Adam(transformer.parameters(), lr=0.0001, betas=(0.9, 0.98), eps=1e-9)"
      ],
      "execution_count": 10,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9vMUp5ymW0JR"
      },
      "source": [
        "Colación\n",
        "---------\n",
        "\n",
        "Como se ve en la sección ***Origen y procesamiento de datos***, nuestro iterador de datos produce un par de cadenas sin procesar.\n",
        "Necesitamos convertir estos pares de cadenas en tensores por lotes que puedan ser procesados por nuestra red **Seq2Seq** definido previamente. A continuación, definimos nuestra función de clasificación que convierte un lote de cadenas sin procesar en tensores de lote que se puede introducir directamente en nuestro modelo.  \n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "t3fI-0YDW0JT"
      },
      "source": [
        "from torch.nn.utils.rnn import pad_sequence\n",
        "\n",
        "# función de ayuda para juntar operaciones secuenciales\n",
        "def sequential_transforms(*transforms):\n",
        "    def func(txt_input):\n",
        "        for transform in transforms:\n",
        "            txt_input = transform(txt_input)\n",
        "        return txt_input\n",
        "    return func\n",
        "\n",
        "# función para agregar BOS/EOS y crear tensor para índices de secuencia de entrada\n",
        "def tensor_transform(token_ids: List[int]):\n",
        "    return torch.cat((torch.tensor([BOS_IDX]), \n",
        "                      torch.tensor(token_ids), \n",
        "                      torch.tensor([EOS_IDX])))\n",
        "\n",
        "# El texto de los idiomas src y tgt se transforma para convertir cadenas sin \n",
        "# procesar en índices de tensores\n",
        "text_transform = {}\n",
        "for ln in [SRC_LANGUAGE, TGT_LANGUAGE]:\n",
        "    text_transform[ln] = sequential_transforms(token_transform[ln], #Tokenization\n",
        "                                               vocab_transform[ln], #Numericalization\n",
        "                                               tensor_transform) # Add BOS/EOS and create tensor\n",
        "\n",
        "\n",
        "# función para recopilar muestras de datos en tensores por lotes\n",
        "def collate_fn(batch):\n",
        "    src_batch, tgt_batch = [], []\n",
        "    for src_sample, tgt_sample in batch:\n",
        "        src_batch.append(text_transform[SRC_LANGUAGE](src_sample.rstrip(\"\\n\")))\n",
        "        tgt_batch.append(text_transform[TGT_LANGUAGE](tgt_sample.rstrip(\"\\n\")))\n",
        "\n",
        "    src_batch = pad_sequence(src_batch, padding_value=PAD_IDX)\n",
        "    tgt_batch = pad_sequence(tgt_batch, padding_value=PAD_IDX)\n",
        "    return src_batch, tgt_batch"
      ],
      "execution_count": 11,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HmHg8dPfW0JU"
      },
      "source": [
        "Definamos el ciclo de entrenamiento y evaluación que se llamará para cada época.\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ss2pvLd2W0JV"
      },
      "source": [
        "from torch.utils.data import DataLoader\n",
        "\n",
        "def train_epoch(model, optimizer):\n",
        "    model.train()\n",
        "    losses = 0\n",
        "    train_iter = Multi30k(split='train', language_pair=(SRC_LANGUAGE, TGT_LANGUAGE))\n",
        "    train_dataloader = DataLoader(train_iter, batch_size=BATCH_SIZE, collate_fn=collate_fn)\n",
        "\n",
        "    idx = 0\n",
        "    for src, tgt in train_dataloader:\n",
        "        src = src.to(DEVICE)\n",
        "        tgt = tgt.to(DEVICE)\n",
        "\n",
        "        tgt_input = tgt[:-1, :]\n",
        "\n",
        "        src_mask, tgt_mask, src_padding_mask, tgt_padding_mask = create_mask(src, tgt_input)\n",
        "\n",
        "        logits = model(src, tgt_input, src_mask, tgt_mask,src_padding_mask, tgt_padding_mask, src_padding_mask)\n",
        "\n",
        "        optimizer.zero_grad()\n",
        "\n",
        "        tgt_out = tgt[1:, :]\n",
        "        loss = loss_fn(logits.reshape(-1, logits.shape[-1]), tgt_out.reshape(-1))\n",
        "\n",
        "        print(f\"Iteración {idx}, Loss: {loss.item()}\", end=\"\\n\")\n",
        "        idx += 1\n",
        "\n",
        "        loss.backward()\n",
        "\n",
        "        optimizer.step()\n",
        "        losses += loss.item()\n",
        "\n",
        "    return losses / len(train_dataloader)\n",
        "\n",
        "\n",
        "def evaluate(model):\n",
        "    model.eval()\n",
        "    losses = 0\n",
        "\n",
        "    val_iter = Multi30k(split='valid', language_pair=(SRC_LANGUAGE, TGT_LANGUAGE))\n",
        "    val_dataloader = DataLoader(val_iter, batch_size=BATCH_SIZE, collate_fn=collate_fn)\n",
        "\n",
        "    for src, tgt in val_dataloader:\n",
        "        src = src.to(DEVICE)\n",
        "        tgt = tgt.to(DEVICE)\n",
        "\n",
        "        tgt_input = tgt[:-1, :]\n",
        "\n",
        "        src_mask, tgt_mask, src_padding_mask, tgt_padding_mask = create_mask(src, tgt_input)\n",
        "\n",
        "        logits = model(src, tgt_input, src_mask, tgt_mask,src_padding_mask, tgt_padding_mask, src_padding_mask)\n",
        "        \n",
        "        tgt_out = tgt[1:, :]\n",
        "        loss = loss_fn(logits.reshape(-1, logits.shape[-1]), tgt_out.reshape(-1))\n",
        "        losses += loss.item()\n",
        "\n",
        "    return losses / len(val_dataloader)"
      ],
      "execution_count": 12,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PeiomL6FW0JW"
      },
      "source": [
        "Ahora tenemos todos los ingredientes para entrenar nuestro modelo. ¡Vamos a hacerlo!\n",
        "\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GEIgtBZQW0JW",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "e7004ff1-7f18-4a54-b016-d375bb044b40"
      },
      "source": [
        "from timeit import default_timer as timer\n",
        "NUM_EPOCHS = 1 # 18\n",
        "\n",
        "for epoch in range(1, NUM_EPOCHS+1):\n",
        "    start_time = timer()\n",
        "    train_loss = train_epoch(transformer, optimizer)\n",
        "    end_time = timer()\n",
        "    val_loss = evaluate(transformer)\n",
        "    print((f\"Epoch: {epoch}, Train loss: {train_loss:.3f}, Val loss: {val_loss:.3f}, \"f\"Epoch time = {(end_time - start_time):.3f}s\"))\n"
      ],
      "execution_count": 18,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Iteración 0, Loss: 8.614317893981934\n",
            "Iteración 1, Loss: 8.511731147766113\n",
            "Iteración 2, Loss: 8.387066841125488\n",
            "Iteración 3, Loss: 8.353707313537598\n",
            "Iteración 4, Loss: 8.265826225280762\n",
            "Iteración 5, Loss: 8.16429615020752\n",
            "Iteración 6, Loss: 8.128314971923828\n",
            "Iteración 7, Loss: 8.022416114807129\n",
            "Iteración 8, Loss: 7.930777549743652\n",
            "Iteración 9, Loss: 7.9285502433776855\n",
            "Iteración 10, Loss: 7.8744096755981445\n",
            "Iteración 11, Loss: 7.823192596435547\n",
            "Iteración 12, Loss: 7.763255596160889\n",
            "Iteración 13, Loss: 7.682346343994141\n",
            "Iteración 14, Loss: 7.613320350646973\n",
            "Iteración 15, Loss: 7.566514492034912\n",
            "Iteración 16, Loss: 7.4617486000061035\n",
            "Iteración 17, Loss: 7.4570631980896\n",
            "Iteración 18, Loss: 7.420221328735352\n",
            "Iteración 19, Loss: 7.314501762390137\n",
            "Iteración 20, Loss: 7.260838508605957\n",
            "Iteración 21, Loss: 7.213036060333252\n",
            "Iteración 22, Loss: 7.116337776184082\n",
            "Iteración 23, Loss: 7.078296184539795\n",
            "Iteración 24, Loss: 7.056265354156494\n",
            "Iteración 25, Loss: 6.93980598449707\n",
            "Iteración 26, Loss: 6.9119648933410645\n",
            "Iteración 27, Loss: 6.870570182800293\n",
            "Iteración 28, Loss: 6.734710693359375\n",
            "Iteración 29, Loss: 6.712361812591553\n",
            "Iteración 30, Loss: 6.721524715423584\n",
            "Iteración 31, Loss: 6.621883869171143\n",
            "Iteración 32, Loss: 6.585938453674316\n",
            "Iteración 33, Loss: 6.465880393981934\n",
            "Iteración 34, Loss: 6.435567855834961\n",
            "Iteración 35, Loss: 6.399659156799316\n",
            "Iteración 36, Loss: 6.275651454925537\n",
            "Iteración 37, Loss: 6.280507564544678\n",
            "Iteración 38, Loss: 6.178964138031006\n",
            "Iteración 39, Loss: 6.210146903991699\n",
            "Iteración 40, Loss: 6.292634963989258\n",
            "Iteración 41, Loss: 6.097578525543213\n",
            "Iteración 42, Loss: 6.0725016593933105\n",
            "Iteración 43, Loss: 6.017319202423096\n",
            "Iteración 44, Loss: 5.982271194458008\n",
            "Iteración 45, Loss: 5.849359512329102\n",
            "Iteración 46, Loss: 5.868719577789307\n",
            "Iteración 47, Loss: 5.984291076660156\n",
            "Iteración 48, Loss: 5.762477397918701\n",
            "Iteración 49, Loss: 5.735447883605957\n",
            "Iteración 50, Loss: 5.717535495758057\n",
            "Iteración 51, Loss: 5.622046947479248\n",
            "Iteración 52, Loss: 5.766310214996338\n",
            "Iteración 53, Loss: 5.657112121582031\n",
            "Iteración 54, Loss: 5.611289978027344\n",
            "Iteración 55, Loss: 5.572592258453369\n",
            "Iteración 56, Loss: 5.579448699951172\n",
            "Iteración 57, Loss: 5.583442211151123\n",
            "Iteración 58, Loss: 5.480404376983643\n",
            "Iteración 59, Loss: 5.412841796875\n",
            "Iteración 60, Loss: 5.408481121063232\n",
            "Iteración 61, Loss: 5.362977027893066\n",
            "Iteración 62, Loss: 5.454615592956543\n",
            "Iteración 63, Loss: 5.417118549346924\n",
            "Iteración 64, Loss: 5.412685871124268\n",
            "Iteración 65, Loss: 5.2425689697265625\n",
            "Iteración 66, Loss: 5.2892560958862305\n",
            "Iteración 67, Loss: 5.2790398597717285\n",
            "Iteración 68, Loss: 5.289495468139648\n",
            "Iteración 69, Loss: 5.267638206481934\n",
            "Iteración 70, Loss: 5.3509392738342285\n",
            "Iteración 71, Loss: 5.3133544921875\n",
            "Iteración 72, Loss: 5.261057376861572\n",
            "Iteración 73, Loss: 5.3067803382873535\n",
            "Iteración 74, Loss: 5.300978660583496\n",
            "Iteración 75, Loss: 5.11824893951416\n",
            "Iteración 76, Loss: 5.157479286193848\n",
            "Iteración 77, Loss: 5.070262432098389\n",
            "Iteración 78, Loss: 5.145084381103516\n",
            "Iteración 79, Loss: 5.254518032073975\n",
            "Iteración 80, Loss: 5.176058769226074\n",
            "Iteración 81, Loss: 4.937540531158447\n",
            "Iteración 82, Loss: 5.012692928314209\n",
            "Iteración 83, Loss: 5.082666873931885\n",
            "Iteración 84, Loss: 5.079029560089111\n",
            "Iteración 85, Loss: 4.968675136566162\n",
            "Iteración 86, Loss: 5.0708327293396\n",
            "Iteración 87, Loss: 4.926103115081787\n",
            "Iteración 88, Loss: 5.05012845993042\n",
            "Iteración 89, Loss: 4.878450870513916\n",
            "Iteración 90, Loss: 5.1771368980407715\n",
            "Iteración 91, Loss: 4.959369659423828\n",
            "Iteración 92, Loss: 5.108414173126221\n",
            "Iteración 93, Loss: 4.920158863067627\n",
            "Iteración 94, Loss: 4.895730018615723\n",
            "Iteración 95, Loss: 4.99308967590332\n",
            "Iteración 96, Loss: 4.9274163246154785\n",
            "Iteración 97, Loss: 4.924566745758057\n",
            "Iteración 98, Loss: 4.838410377502441\n",
            "Iteración 99, Loss: 4.927896499633789\n",
            "Iteración 100, Loss: 4.964053153991699\n",
            "Iteración 101, Loss: 4.7607269287109375\n",
            "Iteración 102, Loss: 4.7923808097839355\n",
            "Iteración 103, Loss: 4.940262794494629\n",
            "Iteración 104, Loss: 4.9369001388549805\n",
            "Iteración 105, Loss: 4.90541934967041\n",
            "Iteración 106, Loss: 4.724029064178467\n",
            "Iteración 107, Loss: 4.814856052398682\n",
            "Iteración 108, Loss: 4.954456329345703\n",
            "Iteración 109, Loss: 4.774842262268066\n",
            "Iteración 110, Loss: 4.799868106842041\n",
            "Iteración 111, Loss: 4.734394550323486\n",
            "Iteración 112, Loss: 4.813517093658447\n",
            "Iteración 113, Loss: 4.734835624694824\n",
            "Iteración 114, Loss: 4.88778018951416\n",
            "Iteración 115, Loss: 4.777352333068848\n",
            "Iteración 116, Loss: 4.654625415802002\n",
            "Iteración 117, Loss: 4.803333282470703\n",
            "Iteración 118, Loss: 4.779345512390137\n",
            "Iteración 119, Loss: 4.846133708953857\n",
            "Iteración 120, Loss: 4.929535865783691\n",
            "Iteración 121, Loss: 4.873870372772217\n",
            "Iteración 122, Loss: 4.870710372924805\n",
            "Iteración 123, Loss: 4.717590808868408\n",
            "Iteración 124, Loss: 4.849615097045898\n",
            "Iteración 125, Loss: 4.851555347442627\n",
            "Iteración 126, Loss: 4.704437732696533\n",
            "Iteración 127, Loss: 4.827682971954346\n",
            "Iteración 128, Loss: 4.782487869262695\n",
            "Iteración 129, Loss: 4.873950004577637\n",
            "Iteración 130, Loss: 4.733429431915283\n",
            "Iteración 131, Loss: 4.810376167297363\n",
            "Iteración 132, Loss: 4.6171979904174805\n",
            "Iteración 133, Loss: 4.747613430023193\n",
            "Iteración 134, Loss: 4.809566974639893\n",
            "Iteración 135, Loss: 4.805142402648926\n",
            "Iteración 136, Loss: 4.763585090637207\n",
            "Iteración 137, Loss: 4.711939334869385\n",
            "Iteración 138, Loss: 4.591770172119141\n",
            "Iteración 139, Loss: 4.6241679191589355\n",
            "Iteración 140, Loss: 4.578695774078369\n",
            "Iteración 141, Loss: 4.546008586883545\n",
            "Iteración 142, Loss: 4.498443603515625\n",
            "Iteración 143, Loss: 4.577304840087891\n",
            "Iteración 144, Loss: 4.566615581512451\n",
            "Iteración 145, Loss: 4.704200267791748\n",
            "Iteración 146, Loss: 4.577967166900635\n",
            "Iteración 147, Loss: 4.59989595413208\n",
            "Iteración 148, Loss: 4.42252254486084\n",
            "Iteración 149, Loss: 4.496256351470947\n"
          ],
          "name": "stdout"
        },
        {
          "output_type": "error",
          "ename": "KeyboardInterrupt",
          "evalue": "ignored",
          "traceback": [
            "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
            "\u001b[0;32m<ipython-input-18-379d0a53a562>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mepoch\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mNUM_EPOCHS\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      5\u001b[0m     \u001b[0mstart_time\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtimer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 6\u001b[0;31m     \u001b[0mtrain_loss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtrain_epoch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtransformer\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moptimizer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      7\u001b[0m     \u001b[0mend_time\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mtimer\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      8\u001b[0m     \u001b[0mval_loss\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mevaluate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtransformer\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-12-871c1f8830ab>\u001b[0m in \u001b[0;36mtrain_epoch\u001b[0;34m(model, optimizer)\u001b[0m\n\u001b[1;32m     16\u001b[0m         \u001b[0msrc_mask\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtgt_mask\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msrc_padding_mask\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtgt_padding_mask\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcreate_mask\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msrc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtgt_input\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     17\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 18\u001b[0;31m         \u001b[0mlogits\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mmodel\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msrc\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtgt_input\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msrc_mask\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtgt_mask\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0msrc_padding_mask\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtgt_padding_mask\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0msrc_padding_mask\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     19\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     20\u001b[0m         \u001b[0moptimizer\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mzero_grad\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1049\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1050\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1051\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1052\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1053\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-8-dfca82d5804b>\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, src, trg, src_mask, tgt_mask, src_padding_mask, tgt_padding_mask, memory_key_padding_mask)\u001b[0m\n\u001b[1;32m     71\u001b[0m                 memory_key_padding_mask: Tensor):\n\u001b[1;32m     72\u001b[0m         \u001b[0msrc_emb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpositional_encoding\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msrc_tok_emb\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msrc\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 73\u001b[0;31m         \u001b[0mtgt_emb\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpositional_encoding\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtgt_tok_emb\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtrg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     74\u001b[0m         outs = self.transformer(src_emb, tgt_emb, src_mask, tgt_mask, None, \n\u001b[1;32m     75\u001b[0m                                 src_padding_mask, tgt_padding_mask, memory_key_padding_mask)\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1049\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1050\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1051\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1052\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1053\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m<ipython-input-8-dfca82d5804b>\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, token_embedding)\u001b[0m\n\u001b[1;32m     25\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     26\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtoken_embedding\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 27\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdropout\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtoken_embedding\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpos_embedding\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0mtoken_embedding\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msize\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m:\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     28\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     29\u001b[0m \u001b[0;31m# Módulo auxiliar para convertir el tensor de índices de entrada en el tensor\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py\u001b[0m in \u001b[0;36m_call_impl\u001b[0;34m(self, *input, **kwargs)\u001b[0m\n\u001b[1;32m   1049\u001b[0m         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks\n\u001b[1;32m   1050\u001b[0m                 or _global_forward_hooks or _global_forward_pre_hooks):\n\u001b[0;32m-> 1051\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mforward_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1052\u001b[0m         \u001b[0;31m# Do not call functions when jit is used\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1053\u001b[0m         \u001b[0mfull_backward_hooks\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnon_full_backward_hooks\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/modules/dropout.py\u001b[0m in \u001b[0;36mforward\u001b[0;34m(self, input)\u001b[0m\n\u001b[1;32m     56\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     57\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mforward\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minput\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0mTensor\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 58\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0mF\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdropout\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtraining\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0minplace\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     59\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     60\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;32m/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py\u001b[0m in \u001b[0;36mdropout\u001b[0;34m(input, p, training, inplace)\u001b[0m\n\u001b[1;32m   1166\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0mp\u001b[0m \u001b[0;34m<\u001b[0m \u001b[0;36m0.0\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mp\u001b[0m \u001b[0;34m>\u001b[0m \u001b[0;36m1.0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1167\u001b[0m         \u001b[0;32mraise\u001b[0m \u001b[0mValueError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"dropout probability has to be between 0 and 1, \"\u001b[0m \u001b[0;34m\"but got {}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mp\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1168\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0m_VF\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdropout_\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtraining\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0minplace\u001b[0m \u001b[0;32melse\u001b[0m \u001b[0m_VF\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdropout\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minput\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtraining\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1169\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1170\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
            "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "AK_NXe84oL6i"
      },
      "source": [
        "# función para generar secuencia de salida usando algoritmo codicioso\n",
        "def greedy_decode(model, src, src_mask, max_len, start_symbol):\n",
        "    src = src.to(DEVICE)\n",
        "    src_mask = src_mask.to(DEVICE)\n",
        "\n",
        "    memory = model.encode(src, src_mask)\n",
        "    ys = torch.ones(1, 1).fill_(start_symbol).type(torch.long).to(DEVICE)\n",
        "    for i in range(max_len-1):\n",
        "        memory = memory.to(DEVICE)\n",
        "        tgt_mask = (generate_square_subsequent_mask(ys.size(0))\n",
        "                    .type(torch.bool)).to(DEVICE)\n",
        "        out = model.decode(ys, memory, tgt_mask)\n",
        "        out = out.transpose(0, 1)\n",
        "        prob = model.generator(out[:, -1])\n",
        "        _, next_word = torch.max(prob, dim=1)\n",
        "        next_word = next_word.item()\n",
        "\n",
        "        ys = torch.cat([ys,\n",
        "                        torch.ones(1, 1).type_as(src.data).fill_(next_word)], dim=0)\n",
        "        if next_word == EOS_IDX:\n",
        "            break\n",
        "    return ys\n",
        "\n",
        "\n",
        "# función real para traducir la oración de entrada al idioma de destino\n",
        "def translate(model: torch.nn.Module, src_sentence: str):\n",
        "    model.eval()\n",
        "    src = text_transform[SRC_LANGUAGE](src_sentence).view(-1, 1)\n",
        "    num_tokens = src.shape[0]\n",
        "    src_mask = (torch.zeros(num_tokens, num_tokens)).type(torch.bool)\n",
        "    tgt_tokens = greedy_decode(\n",
        "        model,  src, src_mask, max_len=num_tokens + 5, start_symbol=BOS_IDX).flatten()\n",
        "    return \" \".join(vocab_transform[TGT_LANGUAGE].lookup_tokens(list(tgt_tokens.cpu().numpy()))).replace(\"<bos>\", \"\").replace(\"<eos>\", \"\")"
      ],
      "execution_count": 19,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "jhHPH5yRW0JX",
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "outputId": "338946c5-e831-4cdd-cccd-b83aab74cb81"
      },
      "source": [
        "# Un grupo de personas se para frente a un iglú\n",
        "# A group of people stands in front of an igloo \n",
        "print(translate(transformer, \"Eine Gruppe von Menschen steht vor einem Iglu .\"))"
      ],
      "execution_count": 20,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            " A man in a woman in a street . \n"
          ],
          "name": "stdout"
        }
      ]
    }
  ]
}