wordcount.txt

1. Prepare Input

# cat data.txt
crazy crazy fox jumped
crazy fox jumped
fox is fast
fox is smart
dog is smart

2. Invoke pyspark

# export SPARK_HOME=...
# SPARK_HOME/bin/pyspark
Python 2.6.9 (unknown, Sep  9 2014, 15:05:12)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 1.2.0
      /_/

Using Python version 2.6.9 (unknown, Sep  9 2014 15:05:12)
SparkContext available as sc.
>>> sc
<pyspark.context.SparkContext object at 0x10ae02210>
>>> lines = sc.textFile("data.txt", 1)
>>> debuglines = lines.collect();
>>> debuglines
[u'crazy crazy fox jumped', 
 u'crazy fox jumped', 
 u'fox is fast', 
 u'fox is smart', 
 u'dog is smart'
]
>>> words = lines.flatMap(lambda x: x.split(' '))
>>> debugwords = words.collect();
>>> debugwords
[
 u'crazy', 
 u'crazy', 
 u'fox', 
 u'jumped', 
 u'crazy', 
 u'fox', 
 u'jumped', 
 u'fox', 
 u'is', 
 u'fast', 
 u'fox', 
 u'is', 
 u'smart', 
 u'dog', 
 u'is', 
 u'smart'
]
>>> ones = words.map(lambda x: (x, 1))
>>> debugones = ones.collect()
>>> debugones
[
 (u'crazy', 1), 
 (u'crazy', 1), 
 (u'fox', 1), 
 (u'jumped', 1), 
 (u'crazy', 1), 
 (u'fox', 1), 
 (u'jumped', 1), 
 (u'fox', 1), 
 (u'is', 1), 
 (u'fast', 1), 
 (u'fox', 1), 
 (u'is', 1), 
 (u'smart', 1), 
 (u'dog', 1), 
 (u'is', 1), 
 (u'smart', 1)
]
>>> counts = ones.reduceByKey(lambda x, y: x + y)
>>> debugcounts = counts.collect()
>>> debugcounts
[
 (u'crazy', 3), 
 (u'jumped', 2), 
 (u'is', 3), 
 (u'fox', 4), 
 (u'dog', 1), 
 (u'fast', 1), 
 (u'smart', 2)
]
>>>
>>> counts.saveAsTextFile("output")

3. Examine Output

# cat output/part*
(u'crazy', 3)
(u'jumped', 2)
(u'is', 3)
(u'fox', 4)
(u'dog', 1)
(u'fast', 1)
(u'smart', 2)