examples/text_summarization/summarization_evaluation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) Microsoft Corporation.  \n",
    "Licensed under the MIT License. \n",
    "\n",
    "# Summarization Evaluation\n",
    "This notebook explains the metrics commonly used to evaluate text summarization results and how to use the evaluation utilities provided in the repo. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ROUGE\n",
    "Recall-Oriented Understudy for Gisting Evaluation(ROUGE) is a set of metrics for evaluating automatic text summarization and machine translation results. The metrics compare machine-generated summaries or translations against one or multiple reference summaries or translations created by human.  \n",
    "Commonly used ROUGE metrics are ROUGE-1, ROUGE-2, and ROUGE-L\n",
    "* ROUGE-1: Overlap of unigrams (single words) between machine-generated and reference summaries. \n",
    "* ROUGE-2: Overlap of bigrams (two adjacent words) between machine-generated and reference summaries.\n",
    "* ROUGE-L: Longest Common Subsequence (LCS), which doesn't require consecutive matches but in-sequence matches that refect sentence-level structure similarity.  \n",
    "\n",
    "For each metric, recall, precision, and F1 score are computed. \n",
    "\n",
    "**Utilities for computing ROUGE**\n",
    "* `compute_rouge_perl`: The [pyrouge](https://github.com/bheinzerling/pyrouge) package based on the ROUGE package written in perl is the most popular package for computing ROUGE scores. We provide the `compute_rouge_perl` function based on pyrouge. This function supports English only. \n",
    "* `compute_rouge_python`: The [py-rouge](https://pypi.org/project/py-rouge/) package is a Python implementation of the ROUGE metric which produces almost the same results as the perl implemenation. Since it's easier to install than pyrouge and can be extended to other languages, we provide the `compute_rouge_python` function based on py-rouge. Currently, English and Hindi are supported. Supports for other languages will be added on an as-needed basis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "\n",
    "nlp_path = os.path.abspath('../../')\n",
    "if nlp_path not in sys.path:\n",
    "    sys.path.insert(0, nlp_path)\n",
    "    \n",
    "from utils_nlp.eval import compute_rouge_perl, compute_rouge_python"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sample inputs\n",
    "Both `compute_rouge_perl` and `compute_rouge_python` takes lists of candidate summaries and reference summaries as inputs. Alternatively, you can also provide paths to files containing the candidates and references and set the `is_input_files` argument to `True`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "summary_candidates = [\"The stock market is doing well this year.\", \"The movie is very popular.\"]\n",
    "summary_references = [\"The stock market is doing really well in 2019.\", \"The movie is very popular among millennials.\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### compute_rouge_python\n",
    "To use `compute_rouge_python`, you only need to install the Python package `py-rouge` and `nltk`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "python_rouge_scores = compute_rouge_python(cand=summary_candidates, ref=summary_references)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"ROUGE-1: {}\".format(python_rouge_scores[\"rouge-1\"]))\n",
    "print(\"ROUGE-2: {}\".format(python_rouge_scores[\"rouge-2\"]))\n",
    "print(\"ROUGE-L: {}\".format(python_rouge_scores[\"rouge-l\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `compute_rouge_python` function can also support non-English languages. Currently, only Hindi is supported. Support for other languages will be added on an as-needed basis.  \n",
    "Note that the Hindi sample inputs are generated by translation, so they are not perfect, but suffcient for testing.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "summary_candidates_hi = [\n",
    "    \"शेयर बाजार इस साल बहुत अच्छा कर रहा है। 2020 के लिए भी यही उम्मीद है।\",\n",
    "    \"नई फिल्म बहुत लोकप्रिय है।\",\n",
    "]\n",
    "summary_references_hi = [\n",
    "    \"शेयर बाजार 2019 में वास्तव में अच्छा कर रहा है। आशा है कि 2020 भी ऐसा ही होगा।\",\n",
    "    \"फिल्म सदियों के बीच बहुत लोकप्रिय है।\",\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "python_rouge_scores_hi = compute_rouge_python(cand=summary_candidates_hi, ref=summary_references_hi, language=\"hi\")\n",
    "print(\"ROUGE-1: {}\".format(python_rouge_scores_hi[\"rouge-1\"]))\n",
    "print(\"ROUGE-2: {}\".format(python_rouge_scores_hi[\"rouge-2\"]))\n",
    "print(\"ROUGE-L: {}\".format(python_rouge_scores_hi[\"rouge-l\"]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### compute_rouge_perl\n",
    "To use `compute_rouge_perl`, in addition to installing the Python package `pyrouge`, you also need to go through the following setup steps on a Linux machine.  \n",
    "**NOTE**: Set `PYTHON_PATH` to the root directory of the conda environment where you installed `pyrouge` first. You can use \"conda env list\" to find the`PYTHON_PATH`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "python_path = !which python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PYTHON_PATH = \"/\".join(python_path[0].split(os.pathsep)[0].split(\"/\")[0:-2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "git clone https://github.com/andersjo/pyrouge.git\n",
    "PYROUGE_PATH=./pyrouge \n",
    "PYTHON_PATH=$PYTHON_PATH #<root directory of conda environment> #e.g./data/anaconda/envs/nlp_gpu\n",
    "$PYTHON_PATH/bin/pyrouge_set_rouge_path $PYROUGE_PATH/tools/ROUGE-1.5.5\n",
    "\n",
    "# install XML::DOM plugin, instructions https://web.archive.org/web/20171107220839/www.summarizerman.com/post/42675198985/figuring-out-rouge\n",
    "sudo cpan App::cpanminus\n",
    "sudo cpanm XML::DOM\n",
    "\n",
    "# install XLM::Parser and its dependencies\n",
    "sudo apt-get update\n",
    "sudo apt-get install libexpat1-dev -y\n",
    "sudo cpanm  XML::Parser\n",
    "\n",
    "# Fix WordNet issue\n",
    "# Instructions https://web.archive.org/web/20180812011301/http://kavita-ganesan.com/rouge-howto/#IamHavingWordNetExceptions\n",
    "cd  $PYROUGE_PATH/tools/ROUGE-1.5.5/data/\n",
    "rm WordNet-2.0.exc.db\n",
    "\n",
    "cd WordNet-2.0-Exceptions/\n",
    "./buildExeptionDB.pl . exc WordNet-2.0.exc.db\n",
    "cd ..\n",
    "ln -s WordNet-2.0-Exceptions/WordNet-2.0.exc.db WordNet-2.0.exc.db"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "perl_rouge_scores = compute_rouge_perl(cand=summary_candidates, ref=summary_references)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "perl_rouge_scores"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each score, the 95% confidence interval is also computed, i.e. \"\\_cb\" and \"\\_ce\" stand for the beginning  and end of the confidence interval, respectively.  \n",
    "In addition to ROUGE-1, ROUGE-2, ROUGE-L, the perl script computes a few other ROUGE scores. See details of all scores [here](https://en.wikipedia.org/wiki/ROUGE_%28metric%29).  "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (nlp_gpu)",
   "language": "python",
   "name": "nlp_gpu"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}