notebooks/examples/5.1- File IO.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with files\n",
    "\n",
    "Most software needs to interact with the world one way or another. \n",
    "\n",
    "The most important Unix paradigm for programs is a file. In Unix philosophy almost everything is a file and therefore being able to work with files gives you control over many things.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## File I/O\n",
    "\n",
    "To read files, there exists a built-in called ``open``. Open takes a path/filename and a **mode**.\n",
    "\n",
    "In Python, a file can be opened in the following modes\n",
    "* reading, \"r\"\n",
    "* writing, \"w\" (destroys everything in file)\n",
    "* appending, \"a\" (appends to the end of the file\n",
    "\n",
    "By default, a file is opened in **text** mode, that is to say the contents of the file are interpreted as text. If you want to handle the file as binary data, you can open the file in **binary** mode, either \"rb\", \"wb\" and \"ab\" depending on which mode you want to open it.\n",
    "\n",
    "The ``readline()`` function reads a file until a newline \"\\n\" character is reached and returns that string. A file object is iterable so you can also iterate over lines using a ``for`` statement.\n",
    "\n",
    "You should always take care to remember to close any files you've opened."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "filename = \"../data/example_file.txt\"\n",
    "fp = open(filename, \"w\")\n",
    "for string in [\"Hello\", \"Hey\", \"moi\"]:\n",
    "    fp.write(string + \"\\n\")\n",
    "fp.close()\n",
    "\n",
    "fp = open(filename, \"r\")\n",
    "for line in fp:\n",
    "    print(line.strip()) # the \\n is contained in the line, calling strip removes whitespace at the end and beginning\n",
    "fp.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Context handlers\n",
    "\n",
    "Python also has a nifty syntax called a context handler for dealing with among others file-like objects. The context handler takes care of calling the open and close functions and deals with closing the file no matter what happens.\n",
    "\n",
    "It makes for somewhat cleaner syntax as a whole."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(filename, \"r\") as file:\n",
    "    for line in file:\n",
    "        print(line.strip())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## File i/o packages in stdlib\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## Compressed files\n",
    "\n",
    "Python standard library contains modules for dealing with ``gzip`` (.gz), ``bzip2`` (.bz2) and the less used LZMA algorithms. Additionally there is support for opening ZIP and ``tar`` archives, which may contain multiple files. Information can be found [in the documentation](https://docs.python.org/3/library/archiving.html).\n",
    "\n",
    "There are more tools in the Python Package Index for many other formats.\n",
    "\n",
    "The beauty of handling compressed files is that the abstraction level is essentially the same as working with an uncompressed file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gzip \n",
    "# the library gzip offers an API like open(), see https://docs.python.org/3/library/gzip.html\n",
    "\n",
    "zipped_file_name = \"../data/zipped_file.gz\"\n",
    "with gzip.open(zipped_file_name, \"wt\") as zipped_file:\n",
    "    for line in [\"This\", \"is\", \"an\", \"example\", \".\"]:\n",
    "        zipped_file.write((line + \"\\n\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## Go ahead, try to read the lines read from zipped_file_name and print them.\n",
    "## It goes just like in the examples above, except with gzip.open instead of open\n",
    "## as this is text, you'll need to open the file in mode \"rt\" and not just r"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The implementation technical details like the \"rt\" vs \"r\" mode vary a bit, check [documentation](https://docs.python.org/3/library/archiving.html) when unsure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data interchange formats\n",
    "\n",
    "Data can be stored in myriad ways.\n",
    "\n",
    "Very common is the so-called Comma-Separated Values\n",
    "```\n",
    "header1,header\n",
    "1,0\n",
    "0,1\n",
    "1,0\n",
    "```\n",
    "\n",
    "Another common one is JSON\n",
    "\n",
    "```\n",
    "{\"key\": \"value\", \"key2\": \"value2\"}\n",
    "```\n",
    "\n",
    "XML is, of course an alternative\n",
    "\n",
    "```\n",
    "<?xml version=\"1.0\"?>\n",
    "<data>\n",
    "    <country name=\"Liechtenstein\">\n",
    "        <rank>1</rank>\n",
    "        <year>2008</year>\n",
    "        <gdppc>141100</gdppc>\n",
    "        <neighbor name=\"Austria\" direction=\"E\"/>\n",
    "        <neighbor name=\"Switzerland\" direction=\"W\"/>\n",
    "    </country>\n",
    "</data>\n",
    "```\n",
    "XML examples would require such in-depth knowledge of XML that they are not covered in this notebook. Suffice to say that it is possible to handle XML files.\n",
    "\n",
    "\n",
    "Many software packages read their configurations from files in the INI format\n",
    "\n",
    "```\n",
    "[default]\n",
    "value = 5\n",
    "\n",
    "[special_configs]\n",
    "bigger_value = 6\n",
    "```\n",
    "\n",
    "These 4 are mentioned as examples because there are libraries in the Python Standard Library:\n",
    "- [CSV](https://docs.python.org/3/library/csv.html)\n",
    "- [JSON](https://docs.python.org/3/library/json.html)\n",
    "- [XML](https://docs.python.org/3/library/xml.html)\n",
    "- [INI](https://docs.python.org/3/library/configparser.html)\n",
    "\n",
    "Of special interest is also Python's internal [``pickle``](https://docs.python.org/3/library/pickle.html) expressly for the purpose of serializing Python objects."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# we will use this object throughout the examples to illustrate different file format handling\n",
    "pythons = [\n",
    "    {\"name\": \"Graham Chapham\", \"birthyear\": 1941, \"dead\": True},\n",
    "    {\"name\": \"Eric Idle\", \"birthyear\": 1943, \"dead\": False},\n",
    "    {\"name\": \"Terry Gilliam\", \"birthyear\": 1940, \"dead\": False},\n",
    "    {\"name\": \"Terry Jones\", \"birthyear\": 1942, \"dead\": False},\n",
    "    {\"name\": \"John Cleese\", \"birthyear\": 1939, \"dead\": False},\n",
    "    {\"name\": \"Michael Palin\", \"birthyear\": 1939, \"dead\": False},\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### CSV\n",
    "\n",
    "The Comma-separated values format seems deceptively simple at first and a casual reader can be tempted into trying to create a parser themselves.\n",
    "\n",
    "    \"What could possibly go wrong? It's a really simple format after all.\" \n",
    "    - every starting developer at least once in their career\n",
    "    \n",
    "The number of different conventions makes parsing all kinds of CSV files highly nontrivial and it is therefore good that there is a separate library for that purpose.\n",
    "\n",
    "There are two simple ways to use the built-in [``csv``](https://docs.python.org/3/library/csv.html) -library:\n",
    "* with headers\n",
    "* without headers\n",
    "\n",
    "**Without headers**\n",
    "* to read call csv.reader() with an open file\n",
    " * iterate over returned object\n",
    "* to write call csv.writer with an open file\n",
    " * call the writerow() -function of the returned object\n",
    " \n",
    "**With headers**\n",
    "* to read call csv.DictReader() with an open file\n",
    " * iterate over returned object, returned values are ``dict``\n",
    "* to write call csv.DictWriter() with an open file and **fieldnames** as a list\n",
    " * call the writeheader() function to write the header row\n",
    " * call the writerow() function of the returned object with a ``dict```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import csv\n",
    "filename = \"../data/example.csv\"\n",
    "\n",
    "\n",
    "\n",
    "with open(filename, \"w\") as file_:\n",
    "    writer = csv.DictWriter(file_, fieldnames=[\"name\", \"birthyear\", \"dead\"])\n",
    "    writer.writeheader()\n",
    "    for performer in pythons:\n",
    "        writer.writerow(performer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def print_performer_dict(performer):\n",
    "    import datetime\n",
    "    this_year = datetime.datetime.now().year\n",
    "    if performer[\"dead\"].lower() == \"true\":\n",
    "        print(\"%s is dead\" % performer[\"name\"])\n",
    "    else:\n",
    "        print(\"%s turns %d this year\" % (performer[\"name\"], \n",
    "                                         this_year - int(performer[\"birthyear\"])))\n",
    "\n",
    "        \n",
    "with open(filename, \"r\") as file_:\n",
    "    reader = csv.DictReader(file_)\n",
    "    for performer in reader:\n",
    "        print_performer_dict(performer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note** how the truth value and number needed a bit of tinkering. This is one of the downsides of the CSV format, there is no agreed upon way to mark what is a string and what is a number and what is a boolean value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## JSON\n",
    "\n",
    "JSON is a data interchange format of the web age. It has several flaws, like CSV but yet it is widely used.\n",
    "\n",
    "In Python, one can usually convert ``dict``s as JSON hashes and lists as JSON lists, with some minor caveats. JSON doesn't differentiate between lists and sets, and Python only permits immutable objects as dictionary keys. Also, there is no simple agreed upon way to encode dates and times in JSON ([ISO8601](https://en.wikipedia.org/wiki/ISO_8601) for human-readable dates and possibly Unix timestamps for machine readable dates are recommended).\n",
    "\n",
    "Also, the default [json](https://docs.python.org/3/library/json.html) library may not be optimal in many respects. There are alternatives, like\n",
    "* [simplejson](https://pypi.python.org/pypi/simplejson)\n",
    "* [ujson](https://pypi.python.org/pypi/ujson)\n",
    "\n",
    "The different libraries convert corner cases differently and it's usually not a good idea to use JSON as a persistence format between multiple Python softwares.\n",
    "\n",
    "However the requirement does not rise very often when dealing with Internet-based systems.\n",
    "\n",
    "The ``dump``and ``load`` functions operate directly on files and take a filelike object as a parameter. The ``dumps``and ``loads``functions return and read a string, which is what the s stands for."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "# we have two strategies, store the entire object as JSON or store each row as a separate JSON object,\n",
    "# both exist in the wild world so both will be shown\n",
    "# fortunately our dicts only contain very simple datums so there will be no issue\n",
    "ex_1_file = \"../data/example_json_1.json\"\n",
    "ex_2_file = \"../data/example_json_2.json\"\n",
    "\n",
    "with open(ex_1_file, \"w\") as file_:\n",
    "    json.dump(pythons, file_)\n",
    "\n",
    "with open(ex_2_file, \"w\") as file_:\n",
    "    for performer in pythons:\n",
    "        json.dump(performer, file_)\n",
    "        file_.write(\"\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#reading back\n",
    "\n",
    "def print_performer_dict_2(performer):\n",
    "    import datetime\n",
    "    this_year = datetime.datetime.now().year\n",
    "    if performer[\"dead\"]:\n",
    "        print(\"%s is dead\" % performer[\"name\"])\n",
    "    else:\n",
    "        print(\"%s turns %d this year\" % (performer[\"name\"], \n",
    "                                         this_year -performer[\"birthyear\"]))\n",
    "\n",
    "with open(ex_1_file, \"r\") as file_:\n",
    "    data = json.load(file_)\n",
    "    for performer in data:\n",
    "        print_performer_dict_2(performer)\n",
    "print(\"####\")\n",
    "with open(ex_2_file, \"r\") as file_:\n",
    "    for line in file_:\n",
    "        performer = json.loads(line)\n",
    "        print_performer_dict_2(performer)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pickle\n",
    "\n",
    "The simplest way to store Python objects is ``pickle``. It is the standard way to serialize and deserialize Python objects.\n",
    "\n",
    "Pickle serializes Python objects into strings that can be unpickled by other Python processes and threads. It can pickle almost any data presented in Python. The tricky part is ensuring that both Python processes have the same version (pickle is not backwards-compatible) and that they have the same versions of all relevant libraries. \n",
    "\n",
    "Another caveat is that other programming languages don't support ``pickle``, it's Python-only.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "pickled_pythons = pickle.dumps(pythons) #pickle also has dump and dumps like json\n",
    "#we could write pickled_pythons to a file here if we wanted to, but that's not really the point of the exercise\n",
    "unpickled_pythons = pickle.loads(pickled_pythons)\n",
    "\n",
    "print(str(pythons) == str(unpickled_pythons))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The beautiful thing about ``pickle`` is that it will serialize complex objects and deserialize them the same way. \n",
    "\n",
    "For example when training classifiers and regressors in machine learning one can train a classifier on a powerful computer for a long time until the algorithm converges, pickle the resulting object (classifier or regressor) and distribute that to other machines."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## io-module\n",
    "\n",
    "Many Python libraries operate on files or strings. Some library writers assume that everyone will always want to pass a file to their library. Others assume that the results should always be written to a file on the filesystem even when that is not strictly necessary.\n",
    "\n",
    "For that purpose the [io](https://docs.python.org/3/library/io.html) library in Python offers tools to create objects that look like files, even when they aren't.\n",
    "\n",
    "There are two classes\n",
    "* StringIO for personating a file opened in text-mode\n",
    "* BinaryIO for personating a file opened in bytes-mode\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import io\n",
    "\n",
    "my_output = io.StringIO()\n",
    "writer = csv.DictWriter(my_output, fieldnames=[\"name\", \"birthyear\", \"dead\"])\n",
    "writer.writeheader()\n",
    "writer.writerows(pythons)\n",
    "file_contents = my_output.getvalue()\n",
    "print(\"file contents would have been:\\n\")\n",
    "print(file_contents)\n",
    "\n",
    "print(\"---\")\n",
    "#let's construct another StringIO and use csv to read from a string and not a file\n",
    "my_input = io.StringIO(file_contents)\n",
    "reader = csv.DictReader(my_input)\n",
    "for line in reader:\n",
    "    print_performer_dict(line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}