NLP/Notebooks/0_NLP_Basics.ipynb

{
 "metadata": {
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  },
  "orig_nbformat": 4,
  "kernelspec": {
   "name": "python3",
   "display_name": "Python 3.7.6 64-bit ('ml': conda)"
  },
  "interpreter": {
   "hash": "774e39f5f21f7b60062f3abb90b8afe0139135a3792f728ee4bd9a31085203c9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2,
 "cells": [
  {
   "source": [
    "# NLP\n",
    " Book Reference: Practical Natural Language Processing - June 2020\n",
    "\n",
    " <img src=\"images/book.png\">"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "<img src=\"images/h1.png\">"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "<img src=\"images/h2.png\">"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "NLP project pipeline\n",
    "\n",
    "<img src=\"images/Pipeline_NLP.png\">"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "## 1. Data Acquisition\n",
    "a) Use a public dataset  \n",
    "b) Scrape data  \n",
    "c) Product intervention  \n",
    "d) Data augmentation\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "## 2. Text Extraction and Cleanup\n",
    "Process of extracting raw text from the input data by removing all the other non-textual information, such as markup, metadata, etc., and converting the text to the required encoding format.\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "<img src=\"images/text_topy.png\">"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      " \n\n \n\nIn the nineteenth century the only kind of linguistics considered\nseriously was this comparative and historical study of words in languages\nknown or believed to be raguate say the Semitic languages, ot the Indo-\nEuropean languages. It is significant that the Germans who really made\nthe subject what it was, used the term Indo-germanisch. ‘Those who know\nthe popular works of Otto Jespersen will remember how firmly he\ndeclares that linguistic science is historical. And those who have noticed\n\n \n\n \n\f\n"
     ]
    }
   ],
   "source": [
    "from PIL import Image\n",
    "from pytesseract import image_to_string\n",
    "filename = \"images/text_topy.png\"\n",
    "text = image_to_string(Image.open(filename))\n",
    "print(text)"
   ]
  },
  {
   "source": [
    "## 3. Pre-Processing\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "a) Preliminaries: Sentence segmentation and word tokenization.  \n",
    "b) Frequent steps: Stop word removal, stemming and lemmatization, removing digits/punctuation,\n",
    "lowercasing, etc.  \n",
    "c) Other steps: Normalization, language detection, code mixing, transliteration, etc.  \n",
    "d) Advanced processing: POS tagging, parsing, coreference resolution, etc.  "
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "### Sentence Segmentation"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "['In the previous chapter, we saw examples of some common NLP\\napplications that we might encounter in everyday life.', 'If we were asked to\\nbuild such an application, think about how we would approach doing so at our\\norganization.', 'We would normally walk through the requirements and break the\\nproblem down into several sub-problems, then try to develop a step-by-step\\nprocedure to solve them.', 'Since language processing is involved, we would also\\nlist all the forms of text processing needed at each step.', 'This step-by-step\\nprocessing of text is known as pipeline.', 'It is the series of steps involved in\\nbuilding any NLP model.', 'These steps are common in every NLP project, so it\\nmakes sense to study them in this chapter.', 'Understanding some common procedures\\nin any NLP pipeline will enable us to get started on any NLP problem encountered\\nin the workplace.', 'Laying out and developing a text-processing pipeline is seen\\nas a starting point for any NLP application development process.', 'In this\\nchapter, we will learn about the various steps involved and how they play\\nimportant roles in solving the NLP problem and we’ll see a few guidelines\\nabout when and how to use which step.', 'In later chapters, we’ll discuss\\nspecific pipelines for various NLP tasks (e.g., Chapters 4–7).']\n"
     ]
    }
   ],
   "source": [
    "from nltk.tokenize import sent_tokenize, word_tokenize\n",
    "mytext = \"\"\"In the previous chapter, we saw examples of some common NLP\n",
    "applications that we might encounter in everyday life. If we were asked to\n",
    "build such an application, think about how we would approach doing so at our\n",
    "organization. We would normally walk through the requirements and break the\n",
    "problem down into several sub-problems, then try to develop a step-by-step\n",
    "procedure to solve them. Since language processing is involved, we would also\n",
    "list all the forms of text processing needed at each step. This step-by-step\n",
    "processing of text is known as pipeline. It is the series of steps involved in\n",
    "building any NLP model. These steps are common in every NLP project, so it\n",
    "makes sense to study them in this chapter. Understanding some common procedures\n",
    "in any NLP pipeline will enable us to get started on any NLP problem encountered\n",
    "in the workplace. Laying out and developing a text-processing pipeline is seen\n",
    "as a starting point for any NLP application development process. In this\n",
    "chapter, we will learn about the various steps involved and how they play\n",
    "important roles in solving the NLP problem and we’ll see a few guidelines\n",
    "about when and how to use which step. In later chapters, we’ll discuss\n",
    "specific pipelines for various NLP tasks (e.g., Chapters 4–7).\"\"\"\n",
    "my_sentences = sent_tokenize(mytext)\n",
    "print(my_sentences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "In the previous chapter, we saw examples of some common NLP\napplications that we might encounter in everyday life.\n['In', 'the', 'previous', 'chapter', ',', 'we', 'saw', 'examples', 'of', 'some', 'common', 'NLP', 'applications', 'that', 'we', 'might', 'encounter', 'in', 'everyday', 'life', '.']\nIf we were asked to\nbuild such an application, think about how we would approach doing so at our\norganization.\n['If', 'we', 'were', 'asked', 'to', 'build', 'such', 'an', 'application', ',', 'think', 'about', 'how', 'we', 'would', 'approach', 'doing', 'so', 'at', 'our', 'organization', '.']\n"
     ]
    }
   ],
   "source": [
    "for sentence in my_sentences[0:2]:\n",
    "    print(sentence)\n",
    "    print(word_tokenize(sentence))"
   ]
  },
  {
   "source": [
    "### Frequent Steps\n",
    "Common Words"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "['This', 'is', 'a', 'sample', 'sentence', ',', 'showing', 'off', 'the', 'stop', 'words', 'filtration', '.']\n['This', 'sample', 'sentence', ',', 'showing', 'stop', 'words', 'filtration', '.']\n"
     ]
    }
   ],
   "source": [
    "example_sent = \"\"\"This is a sample sentence,\n",
    "                  showing off the stop words filtration.\"\"\"\n",
    "stop_words = set(stopwords.words(\"english\"))\n",
    "word_tokens = word_tokenize(example_sent)\n",
    "filtered_sentence = [w for w in word_tokens if not w.lower() in stop_words]\n",
    "filtered_sentence = []\n",
    " \n",
    "for w in word_tokens:\n",
    "    if w not in stop_words:\n",
    "        filtered_sentence.append(w)\n",
    " \n",
    "print(word_tokens)\n",
    "print(filtered_sentence)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "['Esta', 'es', 'una', 'muestra', 'de', 'stop', 'words']\n['Esta', 'muestra', 'stop', 'words']\n"
     ]
    }
   ],
   "source": [
    "example_sent = \"\"\"Esta es una muestra de stop words\"\"\"\n",
    "stop_words = set(stopwords.words(\"spanish\"))\n",
    "word_tokens = word_tokenize(example_sent)\n",
    "filtered_sentence = [w for w in word_tokens if not w.lower() in stop_words]\n",
    "filtered_sentence = []\n",
    " \n",
    "for w in word_tokens:\n",
    "    if w not in stop_words:\n",
    "        filtered_sentence.append(w)\n",
    " \n",
    "print(word_tokens)\n",
    "print(filtered_sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "{'habríais', 'quien', 'estuvisteis', 'tuvierais', 'fuisteis', 'eres', 'un', 'tuvieses', 'tendremos', 'en', 'tendrían', 'sí', 'y', 'otra', 'tuvimos', 'habíamos', 'hubisteis', 'hubo', 'sentidas', 'contra', 'de', 'le', 'fuerais', 'hubimos', 'tenidos', 'siente', 'estará', 'tengamos', 'estando', 'tuviesen', 'hubieron', 'otro', 'fueran', 'míos', 'otras', 'sentida', 'ellas', 'las', 'tendrá', 'del', 'esos', 'suya', 'tenga', 'otros', 'teníais', 'poco', 'tuviésemos', 'hemos', 'seas', 'tendré', 'tu', 'están', 'estás', 'desde', 'vosotros', 'hayáis', 'estéis', 'estuviesen', 'mío', 'seamos', 'la', 'estuviésemos', 'estaba', 'nuestras', 'estaría', 'mucho', 'una', 'mis', 'tuvieran', 'tuvieseis', 'han', 'eras', 'fuésemos', 'tenían', 'estén', 'estuviese', 'hubieras', 'yo', 'habéis', 'lo', 'qué', 'donde', 'estaríamos', 'sentidos', 'esa', 'ellos', 'estado', 'éramos', 'todo', 'estuviera', 'fueseis', 'todos', 'sean', 'tienen', 'estemos', 'estuviste', 'estad', 'e', 'son', 'serías', 'estáis', 'seréis', 'hubiéramos', 'estados', 'hubieses', 'cual', 'algunos', 'tuyos', 'tendrán', 'habrá', 'habrás', 'estaban', 'él', 'mía', 'esté', 'tuyas', 'tuvieron', 'será', 'tenías', 'o', 'estuvo', 'ha', 'fueron', 'tanto', 'ni', 'quienes', 'que', 'tenida', 'habían', 'habríamos', 'fui', 'tuviéramos', 'ti', 'tuviera', 'vuestro', 'como', 'sintiendo', 'hayas', 'tendría', 'erais', 'ese', 'estaremos', 'hasta', 'habré', 'ella', 'ya', 'es', 'estarás', 'sois', 'tenido', 'también', 'hayan', 'habida', 'tú', 'tuvisteis', 'serás', 'estar', 'ante', 'hubierais', 'tendríamos', 'nada', 'fueras', 'tienes', 'tuve', 'tenéis', 'tuvo', 'tengo', 'tus', 'hubiesen', 'sea', 'muy', 'estaréis', 'algo', 'hayamos', 'esas', 'esta', 'nuestra', 'el', 'pero', 'hay', 'tendríais', 'estuviéramos', 'habrían', 'por', 'estuvieseis', 'está', 'estarías', 'los', 'mi', 'antes', 'has', 'hubiésemos', 'más', 'mí', 'tuviste', 'fuimos', 'tengan', 'tuyo', 'fuéramos', 'estadas', 'a', 'uno', 'me', 'tengáis', 'habrías', 'seremos', 'habíais', 'estos', 'teniendo', 'hubiste', 'tendréis', 'estés', 'fuera', 'estoy', 'mías', 'era', 'hubieseis', 'vuestras', 'habidas', 'habréis', 'muchos', 'habremos', 'vuestros', 'sentid', 'tuviese', 'sus', 'tuvieras', 'hubiese', 'estas', 'estuvimos', 'habrán', 'habías', 'seáis', 'para', 'haya', 'esto', 'os', 'fuiste', 'soy', 'habido', 'eso', 'habría', 'algunas', 'sobre', 'tened', 'nosotras', 'nuestro', 'estabais', 'estuvieron', 'tendrás', 'suyas', 'hube', 'cuando', 'serían', 'sin', 'estuve', 'fueses', 'estabas', 'seríais', 'seré', 'nosotros', 'tenidas', 'suyos', 'eran', 'he', 'porque', 'con', 'estarán', 'estaríais', 'hubieran', 'vosotras', 'tendrías', 'habiendo', 'estuvieses', 'sería', 'les', 'fue', 'había', 'nuestros', 'entre', 'estuvierais', 'seríamos', 'serán', 'estada', 'te', 'este', 'tenemos', 'tengas', 'durante', 'se', 'nos', 'unos', 'suyo', 'vuestra', 'estarían', 'somos', 'fuesen', 'estuvieran', 'no', 'al', 'hubiera', 'tiene', 'su', 'habidos', 'teníamos', 'tenía', 'tuya', 'estábamos', 'estuvieras', 'sentido', 'estamos', 'fuese', 'estaré'}\n"
     ]
    }
   ],
   "source": [
    "print(set(stopwords.words('spanish')))"
   ]
  },
  {
   "source": [
    "### Stemming and Lemmatization\n",
    "\n",
    "Stemming: Stemming refers to the process of removing suffixes and reducing a word to some\n",
    "base form.  \n",
    "\n",
    "Lemmatization: Is the process of mapping all the different forms of a word to its base\n",
    "word, or lemma -> involve linguistic analysis, take longer time to run than stemming , is optional"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "<img src=\"images/stem_lemma.png\">"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "he-->he\nwas-->be\nbetter-->well\nmeeting-->meeting\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "nlp = spacy.load('en_core_web_sm')\n",
    "doc = nlp(u\"he was better meeting\")\n",
    "for token in doc:\n",
    "    print(f'{token}-->{token.lemma_}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "word = fishing\nstemmed_word = fish\nlemma = fishing\n\nword = fishes\nstemmed_word = fish\nlemma = fish\n\nword = fished\nstemmed_word = fish\nlemma = fished\n\n"
     ]
    }
   ],
   "source": [
    "from nltk.stem import WordNetLemmatizer\n",
    "from nltk.stem.snowball import SnowballStemmer\n",
    "\n",
    "lemmatizer = WordNetLemmatizer()\n",
    "stemmer = SnowballStemmer(\"english\")\n",
    "\n",
    "words = [\"fishing\", \"fishes\", \"fished\"]\n",
    "for word in words:\n",
    "    print(f\"word = {word}\")\n",
    "    print(f\"stemmed_word = {stemmer.stem(word)}\")\n",
    "    print(f\"lemma = {lemmatizer.lemmatize(word)}\")\n",
    "    print(\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "word = pensar\nstemmed_word = pens\n\nword = pescando\nstemmed_word = pesc\n\nword = pasado\nstemmed_word = pas\n\n"
     ]
    }
   ],
   "source": [
    "stemmer = SnowballStemmer(\"spanish\")\n",
    "\n",
    "words = [\"pensar\", \"pescando\", \"pasado\"]\n",
    "for word in words:\n",
    "    print(f\"word = {word}\")\n",
    "    print(f\"stemmed_word = {stemmer.stem(word)}\")\n",
    "    #print(f\"lemma = {lemmatizer.lemmatize(word)}\")\n",
    "    print(\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "be\n"
     ]
    }
   ],
   "source": [
    "from nltk.stem import WordNetLemmatizer\n",
    "lemmatizer = WordNetLemmatizer()\n",
    "print(lemmatizer.lemmatize(\"was\")) # verb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "good\n"
     ]
    }
   ],
   "source": [
    "from nltk.stem import WordNetLemmatizer\n",
    "lemmatizer = WordNetLemmatizer()\n",
    "print(lemmatizer.lemmatize(\"better\", pos=\"a\")) # adjective"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "better well\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "sp = spacy.load('en_core_web_sm')\n",
    "token = sp(u'better')\n",
    "for word in token:\n",
    "    print(word.text, word.lemma_)"
   ]
  },
  {
   "source": [
    "<img src=\"images/preproc_steps.png\">"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "## Other Pre-Processing Steps\n",
    "\n",
    "- Text normalization: convert digits to text, expand abbreviations.  \n",
    "- Language detection: We can use the library Polyglot to detect the language of the text."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "## Advanced Processing\n",
    "- POS: Part of Speech Tagging"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "Charles Charles PROPN Xxxxx True False\nSpencer Spencer PROPN Xxxxx True False\nChaplin Chaplin PROPN Xxxxx True False\nwas be AUX xxx True True\nborn bear VERB xxxx True False\non on ADP xx True True\n16 16 NUM dd False False\nApril April PROPN Xxxxx True False\n1889 1889 NUM dddd False False\ntoHannah toHannah PROPN xxXxxxx True False\nChaplin Chaplin PROPN Xxxxx True False\n( ( PUNCT ( False False\nborn bear VERB xxxx True False\nHannah Hannah PROPN Xxxxx True False\nHarriet Harriet PROPN Xxxxx True False\nPedlingham Pedlingham PROPN Xxxxx True False\nHill Hill PROPN Xxxx True False\n) ) PUNCT ) False False\nand and CCONJ xxx True True\nCharles Charles PROPN Xxxxx True False\nChaplin Chaplin PROPN Xxxxx True False\nSr Sr PROPN Xx True False\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "nlp = spacy.load('en_core_web_sm')\n",
    "doc = nlp(u'Charles Spencer Chaplin was born on 16 April 1889 toHannah Chaplin (born Hannah Harriet Pedlingham Hill) and Charles Chaplin Sr')\n",
    "for token in doc:\n",
    "    print(token.text, token.lemma_, token.pos_, token.shape_, token.is_alpha, token.is_stop)"
   ]
  },
  {
   "source": [
    "<img src=\"images/Preproc_pipeline.png\">"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "## 4. Feature Engineering or feature extraction\n",
    "\n",
    "The goal of feature engineering is to capture the characteristics of the text into a numeric\n",
    "vector that can be understood by the ML algorithms. Also called \"text representation\".\n",
    "\n",
    "- Classical NLP/ ML Pipeline : Count words in a review for sentiment analysis task\n",
    "- DL Pipeline: Using word embeddings, vectors representation of words, is difficult to interpret the vector representation."
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "<img src=\"images/feat_engineering.png\">"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "## Text Representation\n",
    "Transform a given text into numerical form so that it can be fed into NLP and ML algorithms.  \n",
    "- Convert images and sound to numeric Representations is straightforward.  \n",
    "- Convert to text to numbers is not straightforward -> 4 categories: \n",
    "1. Basic vectorization approaches  \n",
    "2. Distributed representations  \n",
    "3. Universal language representation  \n",
    "4. Handcrafted features  \n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "## Vector Space Models\n",
    "\n",
    "\n",
    "<img src=\"images/vector_space.png\">\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "### 1. Basic Vectorization Approaches:"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "### One hot Encoding"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "['dog bites man', 'man bites dog', 'dog eats meat', 'man eats food']"
      ]
     },
     "metadata": {},
     "execution_count": 3
    }
   ],
   "source": [
    "documents = [\"Dog bites man.\", \"Man bites dog.\", \"Dog eats meat.\", \"Man eats food.\"]\n",
    "processed_docs = [doc.lower().replace(\".\",\"\") for doc in documents]\n",
    "processed_docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "{'dog': 1, 'bites': 2, 'man': 3, 'eats': 4, 'meat': 5, 'food': 6}\n"
     ]
    }
   ],
   "source": [
    "#Build the vocabulary\n",
    "vocab = {}\n",
    "count = 0\n",
    "for doc in processed_docs:\n",
    "    for word in doc.split():\n",
    "        if word not in vocab:\n",
    "            count = count +1\n",
    "            vocab[word] = count\n",
    "print(vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "#Get one hot representation for any string based on this vocabulary. \n",
    "#If the word exists in the vocabulary, its representation is returned. \n",
    "#If not, a list of zeroes is returned for that word. \n",
    "def get_onehot_vector(somestring):\n",
    "    onehot_encoded = []\n",
    "    for word in somestring.split():\n",
    "        temp = [0]*len(vocab)\n",
    "        if word in vocab:\n",
    "            temp[vocab[word]-1] = 1 # -1 is to take care of the fact indexing in array starts from 0 and not 1\n",
    "        onehot_encoded.append(temp)\n",
    "    return onehot_encoded"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "output_type": "stream",
     "name": "stdout",
     "text": [
      "man bites dog\n[[1, 0, 0, 0, 0, 0], [0, 1, 0, 0, 0, 0], [0, 0, 1, 0, 0, 0]]\n6\n"
     ]
    }
   ],
   "source": [
    "print(processed_docs[1])\n",
    "onehot = get_onehot_vector(processed_docs[0]) #one hot representation for a text from our corpus.\n",
    "print(onehot)\n",
    "print(len(onehot[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "output_type": "execute_result",
     "data": {
      "text/plain": [
       "[[0, 0, 1, 0, 0, 0],\n",
       " [0, 0, 0, 0, 0, 0],\n",
       " [1, 0, 0, 0, 0, 0],\n",
       " [0, 0, 0, 0, 0, 0],\n",
       " [0, 0, 0, 0, 0, 0]]"
      ]
     },
     "metadata": {},
     "execution_count": 18
    }
   ],
   "source": [
    "get_onehot_vector(\"man and dog are good\")"
   ]
  },
  {
   "source": [
    "### Disadvantages\n",
    "- The size of a one-hot vector is directly proportional to size of the vocabulary.  \n",
    "- This representation does not give a fixed-length representation for text.  \n",
    "- It treats words as atomic units and has no notion of (dis)similarity between\n",
    "words. Is very poor at capturing the meaning of the word in relation to other words.  "
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "### Bag of Words\n",
    "See 2_Bag_of_Words.ipynb\n",
    "\n",
    "### Advantages\n",
    "- Like one-hot encoding, BoW is fairly simple to understand and implement.  \n",
    "- The vector space resulting from the BoW scheme captures the semantic similarity of documents. So if two documents have similar vocabulary, they’ll be closer to each other in the vector space and vice versa.\n",
    "- We have a fixed-length encoding for any sentence of arbitrary length.\n",
    "\n",
    "### Disadvatanges  \n",
    "- The size of the vector increases with the size of the vocabulary.  \n",
    "- It does not capture the similarity between different words that mean the same\n",
    "thing. (“I run”, “I ran”,)\n",
    "- As the name indicates, it is a “bag” of words—word order information is lost in\n",
    "this representation"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "### 2. Distributed Representations\n",
    "These methods gained momentum in the past six to seven years. They use neural network architectures to create dense, low-dimensional representations of words and texts.\n",
    "\n",
    "Key ideas:\n",
    "- Distributional representation : BoW, One Hot vector: high dimensional and sparse vectors to represent words\n",
    "- Distributed representation: Word2Vec, Glove: Low Dimensional and dense vectors\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "### 3. Universal Text Representations (State of the art)\n",
    "Use of contextual word representations to obtain word vectors:  \n",
    "- Example : bank  \n",
    "Neural architectures such as recurrent neural networks (RNNs) and transformers\n",
    "were used to develop large-scale models of language (ELMo , BERT), which\n",
    "can be used as pre-trained models to get text representations. Applying transfer learning"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "- Vector Embeddings with Transformers Architectures (BERT)  \n",
    "\n",
    "\n",
    "<img src=\"images/hf_vector1.png\">\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "Each word of a sentence apport in the vectorization of a wor.   \n",
    "<img src=\"images/hf_vector2.png\">\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "A encoder is composed by Bidirectional neural networks and attention mechanisms. \n",
    "\n",
    "\n",
    "<img src=\"images/hf_encoder.png\">\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "source": [
    "ELMO uses Recurrent Neural Networks inside his architecture (LSTM)\n",
    "\n",
    "<img src=\"images/elmo_lstm.png\">\n"
   ],
   "cell_type": "markdown",
   "metadata": {}
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ]
}