Sentiment Analysis Example.ipynb

{
 "metadata": {
  "name": "Sentiment Analysis Example"
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "heading",
     "level": 1,
     "metadata": {},
     "source": [
      "Sentiment Analysis using NLTK"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "In this notebook, I\u2019m going to show you how to build a simple Sentiment Analysis classifier, using a Python framework called Natural Language Toolkit, or nltk. The nltk is an excellent platform for building Python programs to work with human language data. It contains many different modules, providing functionality for most areas of Natural Language Processing. I\u2019m only going to use a very small part of nltk tonight, and I highly recommend that you visit their website at nltk.org, and read the nltk book at [nltk.org/book](http://nltk.org/book) if you are interested in this topic. The nltk book is free, very comprehensive, and doesn\u2019t assume any prior knowledge of Natural Language Processing or even Python.\n",
      "\n",
      "We're going to to solve the following problem:\n",
      "\n",
      "\u201cYour company has developed and released an online budgeting tool (SuperBudget), and your company wants to get an daily update about how people are feeling about it - you know that people are discussing it on Twitter, but you want to monitor the overall feeling people have towards the new product - are they happy or angry?\n",
      "\n",
      "You\u2019ve done some research and there seems to be 100\u2019s of Tweets everyday tagged with your app #SuperBudget.\""
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#some imports for later on\n",
      "from pprint import pprint"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 361
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Training data set"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The first step, and probably the most important and sometimes the most difficult step, is to gather training data. To build this system, we\u2019re going to have to show it many examples of positive and negative tweets. The larger the training set, the better, but I would recommend trying to get at least a few thousand of both positive and negative tweets. They don\u2019t all have to talk about your product, but they do need to be related to your domain - for instance in our case, we would want to get our hands on a list of positively and negatively classified tweets. There are a few ways of doing this, including searching for datasets available online, or paying someone to manually classify this training set."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "pos_tweets = [('I love this new Samsung cellphone - its easy to use and looks great', 'positive'),\n",
      "              ('This new cellphone is great', 'positive'),\n",
      "              (\"My new television is amazing, I'm very happy I bought it\", 'positive'),\n",
      "              (\"This is the best software - it's so easy to use\", 'positive'),\n",
      "              ('I am excited about Windows 8', 'positive'),\n",
      "              (\"I was really impressed by the excellect service I received\", 'positive'),\n",
      "]\n",
      "\n",
      "neg_tweets = [('I hate Windows 8', 'negative'),\n",
      "              ('My new cellphone is terrible', 'negative'),              \n",
      "              ('This is the worst software ever. It crashes every time I try to do something', 'negative'),\n",
      "              ('I hate my stupid new car, I am so frustrated', 'negative'),\n",
      "              (\"If this stupid computer crashes again I'm going to throw it out the window\", 'negative'),\n",
      "              (\"I believe 100% that this is the worst piece of software ever!\", 'negative'),\n",
      "]\n",
      "training_data = pos_tweets + neg_tweets"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 362
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Tokenization"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The next step is to start to transform these sentences into a representation that we can use to build our classifier. The first step to do this is to do what is called Tokenization. Although there are exceptions, in Natural Language Processing you can generally think of words as tokens. Can anyone guess what tokenization therefore means? The process of tokenization is basically the process of taking a block of text and breaking it up into tokens, in other words, taking a sentence and breaking it up into words. In English this is generally easy, as you can just split on spaces and get a relatively good split."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from nltk.tokenize import word_tokenize, sent_tokenize\n",
      "def tokenize(tweet): \n",
      "    tweet = sent_tokenize(tweet)\n",
      "    return word_tokenize(\" \".join(tweet))\n",
      "\n",
      "print tokenize('Tokenize this please! This as well.')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "['Tokenize', 'this', 'please', '!', 'This', 'as', 'well', '.']\n"
       ]
      }
     ],
     "prompt_number": 363
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Normalization"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The next step is called Normalization. This step is important because we want to remove the amount of irrelevant information. In NLP, information is expensive, and we don\u2019t want to have our model worry about irrelevant information. There are alot of ideas that you can use for normalization, but the general idea is that you want to chop out irrelevant information, and keep relevant information. This is also generally a part of the process where experimentation can pay off. Some ideas of normalization include:\n",
      "\n",
      " * Convert all characters to lower case\n",
      " * Remove any punctuation\n",
      " * Remove common words, called stopwords (words like is, are, this, etc)\n",
      " * Remove too short words\n",
      " * Convert numbers to words\n",
      " * Any other idea that you can think of to remove irrelevant information\n",
      "\n",
      "In our program below, you can see that nltk provides us with a list of common english stopwords. We\u2019ll use this list, although it is also sometimes useful to use your own list of handpicked stopwords. We\u2019re also decided to convert everything to lower case (which is a good idea most of the time), and remove short words. Again, this is something that you should experiment with when building your own classifier, as what works depends heavily on the domain you are working in.\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from nltk.corpus import stopwords\n",
      "swords = stopwords.words('english')\n",
      "#print \"Stopwords:\"\n",
      "#print \"----------\"\n",
      "#print swords\n",
      "\n",
      "def normalize(tweet):\n",
      "    \n",
      "    #convert all words to lower case    \n",
      "    tweet = [word.lower() for word in tweet]\n",
      "    \n",
      "    #remove stopwords    \n",
      "    tweet = [word for word in tweet if word not in swords]\n",
      "    \n",
      "    #remove short words\n",
      "    tweet = [word for word in tweet if len(word)>3]\n",
      "    \n",
      "    return tweet\n",
      "\n",
      "print normalize(['NorMalize', 'this', 'Please'])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "['normalize', 'please']\n"
       ]
      }
     ],
     "prompt_number": 364
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Test this pipeline"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Before we go further, let\u2019s look at what this looks like at the moment. To do this, I have written the following function (process_pipeline) to apply the functions in the pipeline to each of the tweets. Don\u2019t worry too much about the code, it just assures that the function can work with both classified and unclassified tweets.\n",
      "\n",
      "Setting up our pipeline and running it against the training set gives the output below. You can see that the tokenization and normalization have helped to remove some irrelevant information, and you might start seeing the data becoming easier for a computer to process."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def process_pipeline(inpt, pipeline, debug=False):\n",
      "    \"\"\"responsible for processing each input item through the pipeline\"\"\"\n",
      "    for tweet in inpt:\n",
      "        #we want this function to work with both classified and unclassified tweets\n",
      "        try:\n",
      "            tweet, cls = tweet\n",
      "        except:\n",
      "            cls = None\n",
      "        if debug: print \"Starting to process '{0}' through pipeline\".format(tweet)\n",
      "        for func in pipeline:            \n",
      "            tweet = func(tweet)\n",
      "            if debug: \n",
      "                print \"processed using pipeline function '{0}' and received output:\".format(func.__name__)\n",
      "                pprint(tweet)\n",
      "        if cls:\n",
      "            yield tweet, cls\n",
      "        else:\n",
      "            yield tweet\n",
      "        if debug: print\n",
      "\n",
      "pipeline = [tokenize, normalize]    \n",
      "processed = list(process_pipeline(training_data, pipeline))\n",
      "pprint(processed)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[(['love', 'samsung', 'cellphone', 'easy', 'looks', 'great'], 'positive'),\n",
        " (['cellphone', 'great'], 'positive'),\n",
        " (['television', 'amazing', 'happy', 'bought'], 'positive'),\n",
        " (['best', 'software', 'easy'], 'positive'),\n",
        " (['excited', 'windows'], 'positive'),\n",
        " (['really', 'impressed', 'excellect', 'service', 'received'], 'positive'),\n",
        " (['hate', 'windows'], 'negative'),\n",
        " (['cellphone', 'terrible'], 'negative'),\n",
        " (['worst', 'software', 'ever.', 'crashes', 'every', 'time', 'something'],\n",
        "  'negative'),\n",
        " (['hate', 'stupid', 'frustrated'], 'negative'),\n",
        " (['stupid', 'computer', 'crashes', 'going', 'throw', 'window'], 'negative'),\n",
        " (['believe', 'worst', 'piece', 'software', 'ever'], 'negative')]\n"
       ]
      }
     ],
     "prompt_number": 365
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Feature extraction"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now that we have cleaned up the data, the next step is to transform it into a format that the algorithms that do the classification can understand. This process is called feature extraction. I won\u2019t go into detail as to why this step is called that, but it relates to terminology used in the field of machine learning.\n",
      "\n",
      "In order for us to use the classification algorithm, we need to convert each Tweet into a dictionary, with keys for each word in the entire set of training data, and value equal to True if that words is contained in the specific Tweet."
     ]
    },
    {
     "cell_type": "heading",
     "level": 3,
     "metadata": {},
     "source": [
      "Feature extraction 1 - First make a list of all possible words in training data"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The first step is to extract all the words from the training set. To do this we setup up our text cleaning pipeline, and then use this word_set function to extract a set of all the words in all of the tweets."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def word_set(tweets):\n",
      "    \"\"\"responsible for extracting a set of all words used in all tweets in the training set\"\"\"\n",
      "\n",
      "    def get_all_words(tweets):    \n",
      "        \"\"\"iterates through all the tweets, and yields each word\"\"\"\n",
      "        for tweet, cls in tweets:\n",
      "            for word in tweet:\n",
      "                yield word\n",
      "\n",
      "    all_words = get_all_words(tweets)    \n",
      "    #deduplicate the words\n",
      "    all_words = set(all_words) \n",
      "    return sorted(all_words) #sort it to ensure consistency between runs\n",
      "\n",
      "\n",
      "text_cleaning_pipeline = [tokenize, normalize]\n",
      "word_set = word_set(process_pipeline(training_data, text_cleaning_pipeline))\n",
      "print \"Word set (all words in all tweets in training set)\"\n",
      "print \"--------------------------------------------------\"\n",
      "print word_set\n",
      "    "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Word set (all words in all tweets in training set)\n",
        "--------------------------------------------------\n",
        "['amazing', 'believe', 'best', 'bought', 'cellphone', 'computer', 'crashes', 'easy', 'ever', 'ever.', 'every', 'excellect', 'excited', 'frustrated', 'going', 'great', 'happy', 'hate', 'impressed', 'looks', 'love', 'piece', 'really', 'received', 'samsung', 'service', 'software', 'something', 'stupid', 'television', 'terrible', 'throw', 'time', 'window', 'windows', 'worst']\n"
       ]
      }
     ],
     "prompt_number": 366
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Feature extraction 2 - Write function to map a tweet to the list of all possible words"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now we are going to use this set to convert each tweet into a representation that we can use to train our classifier. This function extract_features will be the next function in our final pipeline, and its responsible for converting the list of words of each tweet into this dictionary representation that we just discussed. I\u2019ve called the keys contains(<word>) just to make it clear what the value stands for, but the actual key value doesn\u2019t matter that much."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "def extract_features(tweet):\n",
      "    feature_set = {}\n",
      "    for word in word_set: #iterate through the whole set of words\n",
      "        feature_set['contains({0})'.format(word)] = word in tweet\n",
      "    return feature_set\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 367
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Putting it all together"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "Now we have everything setup. As example, let\u2019s look at how our training data is transformed using our final pipeline."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "test_tweet_index = 4\n",
      "\n",
      "final_pipeline = text_cleaning_pipeline + [extract_features]\n",
      "processed_data = list(process_pipeline([training_data[test_tweet_index]], final_pipeline, debug=True))\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Starting to process 'I am excited about Windows 8' through pipeline\n",
        "processed using pipeline function 'tokenize' and received output:\n",
        "['I', 'am', 'excited', 'about', 'Windows', '8']\n",
        "processed using pipeline function 'normalize' and received output:\n",
        "['excited', 'windows']\n",
        "processed using pipeline function 'extract_features' and received output:\n",
        "{'contains(amazing)': False,\n",
        " 'contains(believe)': False,\n",
        " 'contains(best)': False,\n",
        " 'contains(bought)': False,\n",
        " 'contains(cellphone)': False,\n",
        " 'contains(computer)': False,\n",
        " 'contains(crashes)': False,\n",
        " 'contains(easy)': False,\n",
        " 'contains(ever)': False,\n",
        " 'contains(ever.)': False,\n",
        " 'contains(every)': False,\n",
        " 'contains(excellect)': False,\n",
        " 'contains(excited)': True,\n",
        " 'contains(frustrated)': False,\n",
        " 'contains(going)': False,\n",
        " 'contains(great)': False,\n",
        " 'contains(happy)': False,\n",
        " 'contains(hate)': False,\n",
        " 'contains(impressed)': False,\n",
        " 'contains(looks)': False,\n",
        " 'contains(love)': False,\n",
        " 'contains(piece)': False,\n",
        " 'contains(really)': False,\n",
        " 'contains(received)': False,\n",
        " 'contains(samsung)': False,\n",
        " 'contains(service)': False,\n",
        " 'contains(software)': False,\n",
        " 'contains(something)': False,\n",
        " 'contains(stupid)': False,\n",
        " 'contains(television)': False,\n",
        " 'contains(terrible)': False,\n",
        " 'contains(throw)': False,\n",
        " 'contains(time)': False,\n",
        " 'contains(window)': False,\n",
        " 'contains(windows)': True,\n",
        " 'contains(worst)': False}\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 368
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Train classifier"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "The next, and final step in training our classifier, is to train the classifier. Because we are using nltk, this is very easy. There are many different classifying algorithms available, each of them with their own advantages and disadvantages. If you are interested, I recommend you check out the nltk book for more information. For our purposes today, I\u2019m going to use a NaiveBayesClassifier. This classifier uses a very simple algorithm to weight the value of each word in determining the final classification of the text, but its been found to work pretty well especially in text classification problems.\n",
      "\n",
      "Training this classifier with nltk is very easy. We simply setup our pipeline, run the training data through the pipeline, and then use the nltk api to train our classifier. We then use this show_most_informative_features function to give us some insight into how the classifier is going to make its decisions. From this output, you can see that any tweet containing the word cellphone is highly likely to be positive. This of course doesn\u2019t make sense necessarily, and would probably indicate that we need to add some more steps to our data cleaning steps. As the training data size increases this should also start to become less of an issue. \n",
      "\n",
      "We can also see that the classifier is intuitively indicating that tweets without the word \u201cworst\u201d in them are more likely to be positive, and tweets without the word \u201cgreat\u201d in them are more likely to be negative. Looking at this list gives us a pretty good indication of how the classifier views different keywords.\n"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "from nltk import NaiveBayesClassifier\n",
      "final_pipeline = [tokenize, normalize, extract_features]\n",
      "processed = list(process_pipeline(training_data, final_pipeline))\n",
      "\n",
      "classifier = NaiveBayesClassifier.train(processed)\n",
      "classifier.show_most_informative_features(35) \n",
      "#convert to prob (den/den+nom)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Most Informative Features\n",
        "     contains(cellphone) = True           positi : negati =      1.7 : 1.0\n",
        "      contains(software) = True           negati : positi =      1.7 : 1.0\n",
        "         contains(worst) = False          positi : negati =      1.4 : 1.0\n",
        "         contains(great) = False          negati : positi =      1.4 : 1.0\n",
        "          contains(hate) = False          positi : negati =      1.4 : 1.0\n",
        "        contains(stupid) = False          positi : negati =      1.4 : 1.0\n",
        "          contains(easy) = False          negati : positi =      1.4 : 1.0\n",
        "       contains(crashes) = False          positi : negati =      1.4 : 1.0\n",
        "      contains(software) = False          positi : negati =      1.2 : 1.0\n",
        "     contains(cellphone) = False          negati : positi =      1.2 : 1.0\n",
        "         contains(piece) = False          positi : negati =      1.2 : 1.0\n",
        "     contains(impressed) = False          negati : positi =      1.2 : 1.0\n",
        "       contains(believe) = False          positi : negati =      1.2 : 1.0\n",
        "         contains(every) = False          positi : negati =      1.2 : 1.0\n",
        "       contains(service) = False          negati : positi =      1.2 : 1.0\n",
        "    contains(television) = False          negati : positi =      1.2 : 1.0\n",
        "       contains(samsung) = False          negati : positi =      1.2 : 1.0\n",
        "      contains(terrible) = False          positi : negati =      1.2 : 1.0\n",
        "          contains(love) = False          negati : positi =      1.2 : 1.0\n",
        "         contains(throw) = False          positi : negati =      1.2 : 1.0\n",
        "     contains(something) = False          positi : negati =      1.2 : 1.0\n",
        "         contains(happy) = False          negati : positi =      1.2 : 1.0\n",
        "     contains(excellect) = False          negati : positi =      1.2 : 1.0\n",
        "        contains(really) = False          negati : positi =      1.2 : 1.0\n",
        "      contains(computer) = False          positi : negati =      1.2 : 1.0\n",
        "          contains(time) = False          positi : negati =      1.2 : 1.0\n",
        "          contains(best) = False          negati : positi =      1.2 : 1.0\n",
        "         contains(ever.) = False          positi : negati =      1.2 : 1.0\n",
        "          contains(ever) = False          positi : negati =      1.2 : 1.0\n",
        "         contains(going) = False          positi : negati =      1.2 : 1.0\n",
        "      contains(received) = False          negati : positi =      1.2 : 1.0\n",
        "         contains(looks) = False          negati : positi =      1.2 : 1.0\n",
        "        contains(window) = False          positi : negati =      1.2 : 1.0\n",
        "       contains(amazing) = False          negati : positi =      1.2 : 1.0\n",
        "       contains(excited) = False          negati : positi =      1.2 : 1.0\n"
       ]
      }
     ],
     "prompt_number": 369
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Test classifier"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "That\u2019s it, now we have our classifier trained. The last step is to test it. To do this, let\u2019s look back at our test data that we defined originally. In a real life application, you want a much larger set of test data as well, which you can use to test the effectiveness of making various to your program. In this case, let\u2019s ask the system to tell us what it thinks about the various test tweets."
     ]
    },
    {
     "cell_type": "code",
     "collapsed": true,
     "input": [
      "def classify_tweets(tweets):\n",
      "    processed_tweets = list(process_pipeline(tweets, final_pipeline))\n",
      "    for index, tweet in enumerate(processed_tweets):        \n",
      "        tweet_features = [w for w,v in tweet.items() if v]\n",
      "        prob = classifier.prob_classify(tweet)\n",
      "        predict = classifier.classify(tweet)\n",
      "        print \"I'm pretty sure ({0:.2f}%) that the tweet '{1}' is {2} because it {3}\".format(\n",
      "                prob.prob(predict) * 100, tweets[index], predict.upper(), \" and \".join(tweet_features))\n",
      "        print\n",
      "\n",
      "classify_tweets([\n",
      "                 \"I LOVE using #SuperBudget to do my monthly budget - so easy!\",\n",
      "                 \"Impressed with #SuperBudget - its so easy to use even my stupid brother can use it\",\n",
      "                 \"#SuperBudget is the worst, can't get anything done\",\n",
      "                 \"I love how #SuperBudget crashes each time I try to save my budget :(\",                 \n",
      "                 ])\n",
      "\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "I'm pretty sure (97.84%) that the tweet 'I LOVE using #SuperBudget to do my monthly budget - so easy!' is POSITIVE because it contains(love) and contains(easy)\n",
        "\n",
        "I'm pretty sure (86.22%) that the tweet 'Impressed with #SuperBudget - its so easy to use even my stupid brother can use it' is POSITIVE because it contains(impressed) and contains(stupid) and contains(easy)\n",
        "\n",
        "I'm pretty sure (80.36%) that the tweet '#SuperBudget is the worst, can't get anything done' is NEGATIVE because it contains(worst)\n",
        "\n",
        "I'm pretty sure (80.36%) that the tweet 'I love how #SuperBudget crashes each time I try to save my budget :(' is NEGATIVE because it contains(love) and contains(time) and contains(crashes)\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 370
    },
    {
     "cell_type": "heading",
     "level": 2,
     "metadata": {},
     "source": [
      "Further resources"
     ]
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "I hope you enjoyed this look at using NLTK for Sentiment Analysis.\n",
      "\n",
      "NLTK is a great, but if you are interested in Natural Language Processing or Machine Learning, I highly recommend the following libraries as well:\n",
      "\n",
      " * Scikit-learn - A very useful machine learning toolkit that can also be used for Natural Language Processing - http://scikit-learn.org/stable/\n",
      " * Pattern - A useful web mining module containing some tools for doing NLP on web data - http://www.clips.ua.ac.be/pattern"
     ]
    }
   ],
   "metadata": {}
  }
 ]
}