04_Vocabulary_and_Matching.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<small><small><i>\n",
    "All the IPython Notebooks in **[Python Natural Language Processing](https://github.com/milaan9/Python_Python_Natural_Language_Processing)** lecture series by **[Dr. Milaan Parmar](https://www.linkedin.com/in/milaanparmar/)** are available @ **[GitHub](https://github.com/milaan9)**\n",
    "</i></small></small>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "view-in-github"
   },
   "source": [
    "<a href=\"https://colab.research.google.com/github/milaan9/Python_Python_Natural_Language_Processing/blob/main/04_Vocabulary_and_Matching.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uENjwf_oAtNF"
   },
   "source": [
    "# 04 Vocabulary and Matching\n",
    "So far we've seen how a body of text is divided into tokens, and how individual tokens are parsed and tagged with parts of speech, dependencies and lemmas.\n",
    "\n",
    "Lets understand about identify and label specific phrases that match patterns in NLP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nwBXtpFPAtNG"
   },
   "source": [
    "## Rule-based Matching\n",
    "spaCy offers a rule-matching tool called `Matcher` that allows you to build a library of token patterns, then match those patterns against a Doc object to return a list of found matches. You can match on any part of the token including text and annotations, and you can add multiple patterns to the same matcher.\n",
    "For more: https://spacy.io/usage/linguistic-features#section-rule-based-matching"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "3Xd_UfhzAtNH"
   },
   "outputs": [],
   "source": [
    "# Perform standard imports\n",
    "import spacy\n",
    "nlp = spacy.load('en_core_web_sm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "Q-7ylMnzAtNP"
   },
   "outputs": [],
   "source": [
    "# Import the Matcher library\n",
    "from spacy.matcher import Matcher\n",
    "matcher = Matcher(nlp.vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_sqBC_cDAtNU"
   },
   "source": [
    "<font color=green>Here `matcher` is an object that pairs to the current `Vocab` object. We can add and remove specific named matchers to `matcher` as needed.</font>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3mo4HJj1AtNV"
   },
   "source": [
    "### Creating patterns\n",
    "In literature, the phrase 'solar power' might appear as one word or two, with or without a hyphen. In this section we'll develop a matcher named 'SolarPower' that finds all three:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "Az1f4M8EAtNY"
   },
   "outputs": [],
   "source": [
    "pattern1 = [{'LOWER': 'solarpower'}]\n",
    "pattern2 = [{'LOWER': 'solar'}, {'LOWER': 'power'}]\n",
    "pattern3 = [{'LOWER': 'solar'}, {'IS_PUNCT': True}, {'LOWER': 'power'}]\n",
    "\n",
    "matcher.add('SolarPower', None, pattern1, pattern2, pattern3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BszDc5GHAtNd"
   },
   "source": [
    "Let's break this down:\n",
    "* `pattern1` looks for a single token whose lowercase text reads 'solarpower'\n",
    "* `pattern2` looks for two adjacent tokens that read 'solar' and 'power' in that order\n",
    "* `pattern3` looks for three adjacent tokens, with a middle token that can be any punctuation.<font color=green>*</font>\n",
    "\n",
    "<font color=green>\\* Remember that single spaces are not tokenized, so they don't count as punctuation.</font>\n",
    "<br>Once we define our patterns, we pass them into `matcher` with the name 'SolarPower', and set *callbacks* to `None` (more on callbacks later)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "3yYuncGXAtNe"
   },
   "source": [
    "### Applying the matcher to a Doc object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "s0jo-uXbAtNf"
   },
   "outputs": [],
   "source": [
    "doc = nlp(u'The Solar Power industry continues to grow as demand \\\n",
    "for solarpower increases. Solar-power cars are gaining popularity.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HQ32cChlAtNk",
    "outputId": "9e704f1b-2817-43e3-939e-095fe26ddcbd"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(8656102463236116519, 1, 3), (8656102463236116519, 10, 11), (8656102463236116519, 13, 16)]\n"
     ]
    }
   ],
   "source": [
    "found_matches = matcher(doc)\n",
    "print(found_matches)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jdyVZJBXAtNr"
   },
   "source": [
    "`matcher` returns a list of tuples. Each tuple contains an ID for the match, with start & end tokens that map to the span `doc[start:end]`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Sl_b6NtTAtNs",
    "outputId": "7ba9cf1c-29e4-48a8-ca27-37f6aa785940"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "8656102463236116519 SolarPower 1 3 Solar Power\n",
      "8656102463236116519 SolarPower 10 11 solarpower\n",
      "8656102463236116519 SolarPower 13 16 Solar-power\n"
     ]
    }
   ],
   "source": [
    "for match_id, start, end in found_matches:\n",
    "    string_id = nlp.vocab.strings[match_id]  # get string representation\n",
    "    span = doc[start:end]                    # get the matched span\n",
    "    print(match_id, string_id, start, end, span.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "l56-nFNUAtNy"
   },
   "source": [
    "The `match_id` is simply the hash value of the `string_ID` 'SolarPower'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d7i5-S2CAtNz"
   },
   "source": [
    "### Setting pattern options and quantifiers\n",
    "You can make token rules optional by passing an `'OP':'*'` argument. This lets us streamline our patterns list:\n",
    "Ex: Solar+power: here + is OP.To remove this we should use like this"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "adJqxtdaAtN0"
   },
   "outputs": [],
   "source": [
    "# Redefine the patterns:\n",
    "pattern1 = [{'LOWER': 'solarpower'}]\n",
    "pattern2 = [{'LOWER': 'solar'}, {'IS_PUNCT': True, 'OP':'*'}, {'LOWER': 'power'}]\n",
    "\n",
    "# Remove the old patterns to avoid duplication:\n",
    "matcher.remove('SolarPower')\n",
    "\n",
    "# Add the new set of patterns to the 'SolarPower' matcher:\n",
    "matcher.add('SolarPower', None, pattern1, pattern2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "NuBTZbgFAtN5",
    "outputId": "15b90cfc-21bb-4aeb-ef9a-c419bc57bdb5"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(8656102463236116519, 1, 3), (8656102463236116519, 10, 11), (8656102463236116519, 13, 16)]\n"
     ]
    }
   ],
   "source": [
    "found_matches = matcher(doc)\n",
    "print(found_matches)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mxeZlvBGAtN9"
   },
   "source": [
    "This found both two-word patterns, with and without the hyphen!\n",
    "\n",
    "The following quantifiers can be passed to the `'OP'` key:\n",
    "<table><tr><th>OP</th><th>Description</th></tr>\n",
    "\n",
    "<tr ><td><span >\\!</span></td><td>Negate the pattern, by requiring it to match exactly 0 times</td></tr>\n",
    "<tr ><td><span >?</span></td><td>Make the pattern optional, by allowing it to match 0 or 1 times</td></tr>\n",
    "<tr ><td><span >\\+</span></td><td>Require the pattern to match 1 or more times</td></tr>\n",
    "<tr ><td><span >\\*</span></td><td>Allow the pattern to match zero or more times</td></tr>\n",
    "</table>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2S3Vx72bAtN-"
   },
   "source": [
    "### Be careful with lemmas!\n",
    "If we wanted to match on both 'solar power' and 'solar powered', it might be tempting to look for the *lemma* of 'powered' and expect it to be 'power'. This is not always the case! The lemma of the *adjective* 'powered' is still 'powered':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "ApJufoJmAtN_"
   },
   "outputs": [],
   "source": [
    "pattern1 = [{'LOWER': 'solarpower'}]\n",
    "pattern2 = [{'LOWER': 'solar'}, {'IS_PUNCT': True, 'OP':'*'}, {'LEMMA': 'power'}] # CHANGE THIS PATTERN\n",
    "\n",
    "# Remove the old patterns to avoid duplication:\n",
    "matcher.remove('SolarPower')\n",
    "\n",
    "# Add the new set of patterns to the 'SolarPower' matcher:\n",
    "matcher.add('SolarPower', None, pattern1, pattern2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "vcV2pOUbAtOF"
   },
   "outputs": [],
   "source": [
    "doc2 = nlp(u'Solar-powered energy runs solar-powered cars.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HMvZo6AhAtOK",
    "outputId": "a1083168-7c68-48d6-8d7d-30c5013d6a2b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(8656102463236116519, 0, 3), (8656102463236116519, 5, 8)]\n"
     ]
    }
   ],
   "source": [
    "found_matches = matcher(doc2)\n",
    "print(found_matches)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IniRRPXDAtOO"
   },
   "source": [
    "<font color=green>The matcher found the first occurrence because the lemmatizer treated 'Solar-powered' as a verb, but not the second as it considered it an adjective.<br>For this case it may be better to set explicit token patterns.</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "xg6huqwwAtOP"
   },
   "outputs": [],
   "source": [
    "pattern1 = [{'LOWER': 'solarpower'}]\n",
    "pattern2 = [{'LOWER': 'solar'}, {'IS_PUNCT': True, 'OP':'*'}, {'LOWER': 'power'}]\n",
    "pattern3 = [{'LOWER': 'solarpowered'}]\n",
    "pattern4 = [{'LOWER': 'solar'}, {'IS_PUNCT': True, 'OP':'*'}, {'LOWER': 'powered'}]\n",
    "\n",
    "# Remove the old patterns to avoid duplication:\n",
    "matcher.remove('SolarPower')\n",
    "\n",
    "# Add the new set of patterns to the 'SolarPower' matcher:\n",
    "matcher.add('SolarPower', None, pattern1, pattern2, pattern3, pattern4)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "qxZVPb4PAtOV",
    "outputId": "d7f736bc-6766-4c36-d3c4-dbb444647c66"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(8656102463236116519, 0, 3), (8656102463236116519, 5, 8)]\n"
     ]
    }
   ],
   "source": [
    "found_matches = matcher(doc2)\n",
    "print(found_matches)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QCFezAj_AtOa"
   },
   "source": [
    "## Other token attributes\n",
    "Besides lemmas, there are a variety of token attributes we can use to determine matching rules:\n",
    "<table><tr><th>Attribute</th><th>Description</th></tr>\n",
    "\n",
    "<tr ><td><span >`ORTH`</span></td><td>The exact verbatim text of a token</td></tr>\n",
    "<tr ><td><span >`LOWER`</span></td><td>The lowercase form of the token text</td></tr>\n",
    "<tr ><td><span >`LENGTH`</span></td><td>The length of the token text</td></tr>\n",
    "<tr ><td><span >`IS_ALPHA`, `IS_ASCII`, `IS_DIGIT`</span></td><td>Token text consists of alphanumeric characters, ASCII characters, digits</td></tr>\n",
    "<tr ><td><span >`IS_LOWER`, `IS_UPPER`, `IS_TITLE`</span></td><td>Token text is in lowercase, uppercase, titlecase</td></tr>\n",
    "<tr ><td><span >`IS_PUNCT`, `IS_SPACE`, `IS_STOP`</span></td><td>Token is punctuation, whitespace, stop word</td></tr>\n",
    "<tr ><td><span >`LIKE_NUM`, `LIKE_URL`, `LIKE_EMAIL`</span></td><td>Token text resembles a number, URL, email</td></tr>\n",
    "<tr ><td><span >`POS`, `TAG`, `DEP`, `LEMMA`, `SHAPE`</span></td><td>The token's simple and extended part-of-speech tag, dependency label, lemma, shape</td></tr>\n",
    "<tr ><td><span >`ENT_TYPE`</span></td><td>The token's entity label</td></tr>\n",
    "\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "u6qWHdtyAtOa"
   },
   "source": [
    "### Token wildcard\n",
    "You can pass an empty dictionary `{}` as a wildcard to represent **any token**. For example, you might want to retrieve hashtags without knowing what might follow the `#` character:\n",
    ">`[{'ORTH': '#'}, {}]`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qziLAi0OAtOb"
   },
   "source": [
    "___\n",
    "## PhraseMatcher\n",
    "In the above section we used token patterns to perform rule-based matching. An alternative - and often more efficient - method is to match on terminology lists. In this case we use PhraseMatcher to create a Doc object from a list of phrases, and pass that into `matcher` instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "mm6vPOYbAtOc"
   },
   "outputs": [],
   "source": [
    "# Perform standard imports, reset nlp\n",
    "import spacy\n",
    "nlp = spacy.load('en_core_web_sm')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "zB6DaZVsAtOh"
   },
   "outputs": [],
   "source": [
    "from spacy.matcher import PhraseMatcher\n",
    "matcher = PhraseMatcher(nlp.vocab, attr='LOWER')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "4nqGIvuNErly"
   },
   "outputs": [],
   "source": [
    "terms = ['Galaxy Note', 'iPhone 11', 'iPhone XS', 'Google Pixel']\n",
    "patterns = [nlp(text) for text in terms]\n",
    "matcher.add(\"TerminologyList\", None, *patterns)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "8VccJ1kqBiTs",
    "outputId": "436b97f8-5bfb-4039-b9fb-6e5aeb6def65"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(3766102292120407359, 17, 19), (3766102292120407359, 22, 24), (3766102292120407359, 30, 32), (3766102292120407359, 33, 35)]\n"
     ]
    }
   ],
   "source": [
    "# Borrowed from https://daringfireball.net/linked/2019/09/21/patel-11-pro\n",
    "text_doc = nlp(\"Glowing review overall, and some really interesting side-by-side \"\n",
    "               \"photography tests pitting the iPhone 11 Pro against the \"\n",
    "               \"Galaxy Note 10 Plus and last year’s iPhone XS and Google Pixel 3.\") \n",
    "matches = matcher(text_doc)\n",
    "print(matches)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "SSvSV3JkJAXI",
    "outputId": "96c9fa95-2747-4e16-c12c-93746c145e65"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TerminologyList Galaxy Note\n"
     ]
    }
   ],
   "source": [
    "match_id, start, end = matches[1]\n",
    "print(nlp.vocab.strings[match_id], text_doc[start:end])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "sN-0cgHjJAms"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "4_Vocabulary_&_Matching.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}