doc/training/8_Model_Calibration_with_PEST_and_Veneer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Session 8 - Source model calibration using PEST and Veneer\n",
    "\n",
    "PEST is a highly capable system for model calibration, and for sensitivity and uncertainty analysis. PEST is independent of any particular modelling software and modellers have connected Source to PEST in a number of ways.\n",
    "\n",
    "This tutorial looks at specific support in veneer-py for calibration Source models with PEST. Notably, you can describe the PEST job in the notebook and veneer-py will generate the required PEST configuration files.\n",
    "\n",
    "At the time of writing, this functionality supports basic calibration of model parameters to timeseries observations.\n",
    "\n",
    "## Get PEST\n",
    "\n",
    "PEST can be downloaded from [http://www.pesthomepage.org/](http://www.pesthomepage.org/). You want the standard PC PEST for Windows - *not BEOPEST*. PEST is delivered as a zip file - unzip to a directory on your system (ideally a directory without spaces or special characters in the path - something like `C:\\PEST` is good)\n",
    "\n",
    "This tutorial will gloss over a lot of PEST specifics in order to focus on the connection to Source and Veneer. The PEST manual is very comprehensive and it is available from the PEST homepage and included with the software.\n",
    "\n",
    "\n",
    "## Overview\n",
    "\n",
    "- How will it work?\n",
    "- Describing the PEST ‘job'\n",
    "  - Veneer/Source end-point(s)\n",
    "  - Getting the model ‘ready’ for optimisation\n",
    "  - Describing the calibration parameters\n",
    "  - Describing the objective and any observed data\n",
    "  - What veneer-py is taking care of\n",
    "- Running PEST, feedback\n",
    "- Limitations\n",
    "\n",
    "## Which Model?\n",
    "\n",
    "**Note:** This session uses `ExampleProject/CalibrationExample.rsproj`. You are welcome to work with your own model instead, however you will need to change the notebook text at certain points to reflect the names of nodes, links and functions in your model file.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How will it work?\n",
    "\n",
    "PEST requires a series of configuration file that describe the parameter estimation problem, including:\n",
    "\n",
    "* What program to run and what command line arguments to use\n",
    "* A description of one or more text files that PEST can modify in order to create a unique run of the model\n",
    "* Description of one or more results text files that PEST can read to interpret the results of the model run\n",
    "* Optional files that control parallel processing by running multiple copies of the model in parallel.\n",
    "\n",
    "veneer-py has functionality for describing the calibration problem in Python, and then subsequently generating the files expected by PEST. This includes generating a Python script, which becomes the program that PEST invokes. This Python program calls to a running copy of Veneer to execute the Source model and retrieve results.\n",
    "\n",
    "When running PEST in parallel, PEST controls a number of 'slave' processes, each of which is responsible for running one copy of the simulation at a time:\n",
    "\n",
    "![Running PEST on a Source model from Python](Images/Tutorials/Veneer_PEST.png)\n",
    "\n",
    "In this way, you can run a PEST calibration for a Source model that is running in the main Source windows application. You can start multiple copies of Source in order to support parallel calibrations. Alternatively, you can use the Veneer Command Line tool."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setting up the servers\n",
    "\n",
    "As in Tutorial 7, we'll use the command line and set up a number of copies of the server using the `start()` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from veneer.manage import start, create_command_line, kill_all_now\n",
    "import veneer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'E:\\\\temp\\\\veneer_cmd\\\\FlowMatters.Source.VeneerCmd.exe'"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "veneer_install = 'D:\\\\src\\\\projects\\\\Veneer\\\\Compiled\\\\Source 4.1.1.4484 (public version)'\n",
    "source_version = '4.1.1'\n",
    "cmd_directory = 'E:\\\\temp\\\\veneer_cmd'\n",
    "path = create_command_line(veneer_install,source_version,dest=cmd_directory)\n",
    "path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "catchment_project='ExampleProject/CalibrationExample.rsproj'\n",
    "num_copies=20    # Important - set this to be a number ~ the number of CPU cores in your system!\n",
    "first_port=9950"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9950 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\"\n",
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9951 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\"\n",
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9952 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\"\n",
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9953 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\"\n",
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9954 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\"\n",
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9955 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\"\n",
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9956 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\"\n",
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9957 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\"\n",
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9958 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\"\n",
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9959 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\"\n",
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9960 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\"\n",
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9961 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\"\n",
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9962 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\"\n",
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9963 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\"\n",
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9964 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\"\n",
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9965 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\"\n",
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9966 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\"\n",
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9967 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\"\n",
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9968 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\"\n",
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9969 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\"\n",
      "[3] Loading plugins\n",
      "\n",
      "[5] Loading plugins\n",
      "\n",
      "[6] Loading plugins\n",
      "\n",
      "[7] Loading plugins\n",
      "\n",
      "[8] Loading plugins\n",
      "\n",
      "[8] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[8] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[8] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[8] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[8] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[8] Plugins loaded (5/5)\n",
      "\n",
      "[9] Loading plugins\n",
      "\n",
      "[10] Loading plugins\n",
      "\n",
      "[10] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[10] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[10] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[10] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[10] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[10] Plugins loaded (5/5)\n",
      "\n",
      "[11] Loading plugins\n",
      "\n",
      "[12] Loading plugins\n",
      "\n",
      "[12] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[12] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[12] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[12] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[12] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[12] Plugins loaded (5/5)\n",
      "\n",
      "[13] Loading plugins\n",
      "\n",
      "[13] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[13] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[13] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[13] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[13] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[13] Plugins loaded (5/5)\n",
      "\n",
      "[14] Loading plugins\n",
      "\n",
      "[14] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[14] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[14] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[14] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[14] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[14] Plugins loaded (5/5)\n",
      "\n",
      "[15] Loading plugins\n",
      "\n",
      "[15] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[15] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[15] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[15] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[15] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[15] Plugins loaded (5/5)\n",
      "\n",
      "[16] Loading plugins\n",
      "\n",
      "[16] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[16] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[16] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[16] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[16] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[16] Plugins loaded (5/5)\n",
      "\n",
      "[17] Loading plugins\n",
      "\n",
      "[17] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[17] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[17] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[17] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[17] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[17] Plugins loaded (5/5)\n",
      "\n",
      "[18] Loading plugins\n",
      "\n",
      "[18] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[18] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[18] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[18] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[18] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[18] Plugins loaded (5/5)\n",
      "\n",
      "[19] Loading plugins\n",
      "\n",
      "[19] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[19] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[19] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[19] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[19] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[19] Plugins loaded (5/5)\n",
      "\n",
      "[19] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\n",
      "\n",
      "[0] Loading plugins\n",
      "\n",
      "[0] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[0] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[0] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[0] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[0] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[0] Plugins loaded (5/5)\n",
      "\n",
      "[1] Loading plugins\n",
      "\n",
      "[1] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[1] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[1] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[1] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[1] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[1] Plugins loaded (5/5)\n",
      "\n",
      "[2] Loading plugins\n",
      "\n",
      "[2] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[2] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[2] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[2] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[2] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[2] Plugins loaded (5/5)\n",
      "\n",
      "ERROR[3] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "[3] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[3] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[3] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[3] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[3] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[3] Plugins loaded (5/5)\n",
      "\n",
      "[3] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\n",
      "\n",
      "[4] Loading plugins\n",
      "\n",
      "[4] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[4] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[4] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[4] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[4] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[4] Plugins loaded (5/5)\n",
      "\n",
      "[5] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[5] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[5] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[5] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[5] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[5] Plugins loaded (5/5)\n",
      "\n",
      "[6] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[6] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[6] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[6] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[6] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[6] Plugins loaded (5/5)\n",
      "\n",
      "[6] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\n",
      "\n",
      "[7] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[7] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[7] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[7] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[7] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[7] Plugins loaded (5/5)\n",
      "\n",
      "ERROR[8] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "[8] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\n",
      "\n",
      "[9] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[9] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[9] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[9] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[9] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[9] Plugins loaded (5/5)\n",
      "\n",
      "[9] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\n",
      "\n",
      "[10] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\n",
      "\n",
      "[11] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[11] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[11] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[11] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[11] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[11] Plugins loaded (5/5)\n",
      "\n",
      "ERROR[12] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "[12] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\n",
      "\n",
      "[13] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\n",
      "\n",
      "ERROR[14] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "[14] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\n",
      "\n",
      "ERROR[15] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "[15] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\n",
      "\n",
      "ERROR[16] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "[16] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\n",
      "\n",
      "ERROR[17] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "[17] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\n",
      "\n",
      "ERROR[18] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "[18] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\n",
      "\n",
      "ERROR[19] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "ERROR[0] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "[0] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\n",
      "\n",
      "ERROR[1] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "[1] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\n",
      "\n",
      "ERROR[2] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "[2] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\n",
      "\n",
      "ERROR[4] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "[4] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\n",
      "\n",
      "ERROR[5] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "[5] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\n",
      "\n",
      "ERROR[6] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "ERROR[7] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "[7] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\n",
      "\n",
      "ERROR[9] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "ERROR[10] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "ERROR[11] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "[11] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\CalibrationExample.rsproj\n",
      "\n",
      "ERROR[13] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "[15] Loading project\n",
      "\n",
      "[17] Loading project\n",
      "\n",
      "[19] Loading project\n",
      "\n",
      "[2] Loading project\n",
      "\n",
      "[3] Loading project\n",
      "\n",
      "[5] Loading project\n",
      "\n",
      "[9] Loading project\n",
      "\n",
      "[12] Loading project\n",
      "\n",
      "[14] Loading project\n",
      "\n",
      "[18] Loading project\n",
      "\n",
      "[0] Loading project\n",
      "\n",
      "[1] Loading project\n",
      "\n",
      "[4] Loading project\n",
      "\n",
      "[6] Loading project\n",
      "\n",
      "[7] Loading project\n",
      "\n",
      "[8] Loading project\n",
      "\n",
      "[10] Loading project\n",
      "\n",
      "[11] Loading project\n",
      "\n",
      "[13] Loading project\n",
      "\n",
      "[16] Loading project\n",
      "\n",
      "[15] Project Loaded\n",
      "\n",
      "[15] New Project\n",
      "\n",
      "[15] Scenario 1\n",
      "\n",
      "[17] Project Loaded\n",
      "\n",
      "[17] New Project\n",
      "\n",
      "[17] Scenario 1\n",
      "\n",
      "[17] [3:59:39 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[17] [3:59:39 PM] Started Source RESTful Service on port:9967\n",
      "\n",
      "Server 17 on port 9967 is ready\n",
      "[17] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n",
      "[19] Project Loaded\n",
      "\n",
      "[19] New Project\n",
      "\n",
      "[19] Scenario 1\n",
      "\n",
      "[1] Project Loaded\n",
      "\n",
      "[1] New Project\n",
      "\n",
      "[1] Scenario 1\n",
      "\n",
      "[2] Project Loaded\n",
      "\n",
      "[2] New Project\n",
      "\n",
      "[2] Scenario 1\n",
      "\n",
      "[3] Project Loaded\n",
      "\n",
      "[3] New Project\n",
      "\n",
      "[3] Scenario 1\n",
      "\n",
      "[3] [3:59:39 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[3] [3:59:39 PM] Started Source RESTful Service on port:9953\n",
      "\n",
      "Server 3 on port 9953 is ready\n",
      "[3] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n",
      "[5] Project Loaded\n",
      "\n",
      "[5] New Project\n",
      "\n",
      "[5] Scenario 1\n",
      "\n",
      "[6] Project Loaded\n",
      "\n",
      "[6] New Project\n",
      "\n",
      "[6] Scenario 1\n",
      "\n",
      "[8] Project Loaded\n",
      "\n",
      "[8] New Project\n",
      "\n",
      "[8] Scenario 1\n",
      "\n",
      "[9] Project Loaded\n",
      "\n",
      "[9] New Project\n",
      "\n",
      "[9] Scenario 1\n",
      "\n",
      "[11] Project Loaded\n",
      "\n",
      "[11] New Project\n",
      "\n",
      "[11] Scenario 1\n",
      "\n",
      "[12] Project Loaded\n",
      "\n",
      "[12] New Project\n",
      "\n",
      "[12] Scenario 1\n",
      "\n",
      "[13] Project Loaded\n",
      "\n",
      "[13] New Project\n",
      "\n",
      "[13] Scenario 1\n",
      "\n",
      "[14] Project Loaded\n",
      "\n",
      "[14] New Project\n",
      "\n",
      "[14] Scenario 1\n",
      "\n",
      "[14] [3:59:40 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[14] [3:59:40 PM] Started Source RESTful Service on port:9964\n",
      "\n",
      "Server 14 on port 9964 is ready\n",
      "[14] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n",
      "[15] [3:59:39 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[15] [3:59:39 PM] Started Source RESTful Service on port:9965\n",
      "\n",
      "Server 15 on port 9965 is ready\n",
      "[15] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n",
      "[16] Project Loaded\n",
      "\n",
      "[16] New Project\n",
      "\n",
      "[16] Scenario 1\n",
      "\n",
      "[18] Project Loaded\n",
      "\n",
      "[18] New Project\n",
      "\n",
      "[18] Scenario 1\n",
      "\n",
      "[19] [3:59:39 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[19] [3:59:39 PM] Started Source RESTful Service on port:9969\n",
      "\n",
      "Server 19 on port 9969 is ready\n",
      "[19] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n",
      "[0] Project Loaded\n",
      "\n",
      "[0] New Project\n",
      "\n",
      "[0] Scenario 1\n",
      "\n",
      "[1] [3:59:40 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[1] [3:59:40 PM] Started Source RESTful Service on port:9951\n",
      "\n",
      "Server 1 on port 9951 is ready\n",
      "[1] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n",
      "[2] [3:59:39 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[2] [3:59:39 PM] Started Source RESTful Service on port:9952\n",
      "\n",
      "Server 2 on port 9952 is ready\n",
      "[2] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n",
      "[4] Project Loaded\n",
      "\n",
      "[4] New Project\n",
      "\n",
      "[4] Scenario 1\n",
      "\n",
      "[5] [3:59:39 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[5] [3:59:39 PM] Started Source RESTful Service on port:9955\n",
      "\n",
      "Server 5 on port 9955 is ready\n",
      "[5] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n",
      "[6] [3:59:40 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[6] [3:59:40 PM] Started Source RESTful Service on port:9956\n",
      "\n",
      "Server 6 on port 9956 is ready\n",
      "[6] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n",
      "[7] Project Loaded\n",
      "\n",
      "[7] New Project\n",
      "\n",
      "[7] Scenario 1\n",
      "\n",
      "[9] [3:59:40 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[9] [3:59:40 PM] Started Source RESTful Service on port:9959\n",
      "\n",
      "Server 9 on port 9959 is ready\n",
      "[9] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n",
      "[10] Project Loaded\n",
      "\n",
      "[10] New Project\n",
      "\n",
      "[10] Scenario 1\n",
      "\n",
      "[11] [3:59:40 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[11] [3:59:40 PM] Started Source RESTful Service on port:9961\n",
      "\n",
      "Server 11 on port 9961 is ready\n",
      "[11] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n",
      "[12] [3:59:40 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[12] [3:59:40 PM] Started Source RESTful Service on port:9962\n",
      "\n",
      "Server 12 on port 9962 is ready\n",
      "[12] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n",
      "[18] [3:59:40 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[18] [3:59:40 PM] Started Source RESTful Service on port:9968\n",
      "\n",
      "Server 18 on port 9968 is ready\n",
      "[18] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n",
      "[0] [3:59:40 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[0] [3:59:40 PM] Started Source RESTful Service on port:9950\n",
      "\n",
      "Server 0 on port 9950 is ready\n",
      "[0] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n",
      "[4] [3:59:40 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[4] [3:59:40 PM] Started Source RESTful Service on port:9954\n",
      "\n",
      "Server 4 on port 9954 is ready\n",
      "[4] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n",
      "[7] [3:59:40 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[7] [3:59:40 PM] Started Source RESTful Service on port:9957\n",
      "\n",
      "Server 7 on port 9957 is ready\n",
      "[7] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n",
      "[8] [3:59:40 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[8] [3:59:40 PM] Started Source RESTful Service on port:9958\n",
      "\n",
      "Server 8 on port 9958 is ready\n",
      "[8] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n",
      "[13] [3:59:40 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[13] [3:59:40 PM] Started Source RESTful Service on port:9963\n",
      "\n",
      "Server 13 on port 9963 is ready\n",
      "[13] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n",
      "[16] [3:59:40 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[16] [3:59:40 PM] Started Source RESTful Service on port:9966\n",
      "\n",
      "Server 16 on port 9966 is ready\n",
      "[16] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n",
      "[10] [3:59:40 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[10] [3:59:40 PM] Started Source RESTful Service on port:9960\n",
      "\n",
      "Server 10 on port 9960 is ready\n",
      "[10] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n"
     ]
    }
   ],
   "source": [
    "processes,ports = start(catchment_project,\n",
    "                        n_instances=num_copies,\n",
    "                        ports=first_port,\n",
    "                        debug=True,\n",
    "                        veneer_exe=path,\n",
    "                        remote=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "Also as before, we need a copy of the Veneer client for each copy of the server:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "vs = [veneer.Veneer(port=p) for p in ports]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The catchment\n",
    "\n",
    "We haven't used this project in the earlier tutorials. We can query any of the servers for the network (just as we would if Veneer were running within the Source application)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x64879b0>"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAU0AAAEACAYAAAA3NiR2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnXl4VeW18H8rAQwzCSgKiKCC4oAICChVAwhotaIoim0v\nfrdUb1vbKr1toV97DUdvb6t1xK9Sa7UO10orimJFiUNSR2QsiCJSFZEoMiSEeUrW98d+d7ITzsnZ\nZ8q4fs9znuy8+53Oyc46a71rvesVVcUwDMMIR1ZDT8AwDKMpYULTMAwjAUxoGoZhJIAJTcMwjAQw\noWkYhpEAJjQNwzASIJTQFJH1IrJSRFaIyGJXdoaIvO3KnxORDoH6vxCRdSKyRkTGBcoHi8gqEflI\nRO4JlLcRkTmuzTsi0jtw71pXf62ITAmU9xGRRe7ekyLSKtUPwzAMIx5hNc1KIF9Vz1TVYa7sQeDn\nqnoGMA/4OYCInAJcBQwALgLuFxFxbWYDU1W1P9BfRMa78qlAqar2A+4Bbnd95QI3A2cBw4ECEens\n2twG3On62u76MAzDyChhhaZEqdtPVd90168AV7jrS4E5qnpIVdcD64BhInI00FFVl7h6jwGXuesJ\nwKPuei4w2l2PBwpVtVxVtwOFwIXu3mjgaXf9KHB5yPdiGIaRNGGFpgIvi8gSEfmuK3tfRC5111cB\nvdx1T+DzQNsSV9YT2Bgo3+jKarRR1QqgXETyYvUlIl2BMlWtDPTVI+R7MQzDSJqwQnOkqg4Gvg78\nUES+BnwHuEFElgDtgQNpnJfErxKqjmEYRloJ5TxR1S/dzy0iMg8Ypqp34ZnPiEg/4GJXvQQ4NtC8\nlyuLVR5s84WIZAOdVLVUREqA/FptilR1m4h0FpEsp20G+6qBiNjmesNoIahqxpWpuJqmiLTzPeMi\n0h4YB6wWkSNdWRbwK+APrsl8YLLziPcFTgQWq+omPLN7mHMMTQGeC7S51l1PAl5z1wuBsU5A5gJj\nXRlAkauLa+v3dRiqWm+vgoICG68JjmXjNf3x6oswmmZ3YJ7T2FoBT6hqoYj8WERuwFvvfEZVHwFQ\n1Q9E5G/AB8BB4Ada/Y5uAB4BcoAFqvqSK38IeFxE1gHbgMmurzIRuRVY6saJqOcQApgBzHH3V7g+\nDMMwMkpcoamqnwKDopTPAmbFaPMb4DdRypcBp0cp34/nTIrW1yN4gjbavIbXOXnDMIw0YzuC0kx+\nfr6N1wTHsvGa/nj1hdTnWkBDICLa3N+jYRggImhjcAQZhmEY1ZjQNAzDSAATmoZhGAlgQtMwDCMB\nTGgahmEkgAlNwzCMBDChaRiGkQAmNA3DMBLAhKZhGEYCmNA0DMNIABOahmEYCWBC0zAMIwFMaBqG\nYSSACU3DMIwEMKFpGIaRACY0DcMwEsCEppFR2rbthIhUvQyjqRNKaIrIehFZKSIrRGSxKxskIu/4\nZSJylitvJSKPiMgqEXlfRGYE+hnsyj8SkXsC5W1EZI6IrHN99g7cu9bVXysiUwLlfURkkbv3pIiE\nOo7YqF/27dtJQUERBQVFDT0Vw0gLYTXNSiBfVc9U1WGu7DagQFXPBAqA2135JKCNqg4EhgL/ERCC\ns4Gpqtof6C8i4135VKBUVfsB9/h9uWN7bwbOwjtErUBEOgfGv9P1td31YRiGkVHCCk2JUrcS8AVY\nF6DEXSvQXkSygXbAfmCHiBwNdFTVJa7eY8Bl7noC8Ki7nguMdtfjgUJVLXdH9xYCF7p7o4Gn3fWj\nwOUh34uRYYImeU5Ox4aejmGklbAmrQIvi0gF8EdVfRCYBiwUkTvxhOo5ru5cPCH4JdAWmKaq20Vk\nCLAx0OdGoKe77gl8DqCqFSJSLiJ5wXJHCdBTRLoCZapaGeirR9g3bWQW3yQ3jOZIWKE5UlW/FJEj\ngUIR+RC4ErhRVZ8VkSuBh4GxeGb0IeBooCvwhoi8kuC8wngMQnsVZs6cWXWdn5/fbI8WNYyWRHFx\nMcXFxfU+biihqapfup9bRORZYBgwRVVvdOVzReRPrvo1wEtOC9wiIm/hrW2+CRwb6LYX1SZ9ibv3\nhTPrO6lqqYiUAPm12hSp6jYR6SwiWW6cYF+HERSahmE0D2orQJFIpF7GjbumKSLtRKSDu24PjAPe\nwxNw57vyMcA612QDbk3S1R8BrFHVTUC5iAwTL/ZkCvCcazMfuNZdTwJec9cLgbFOQObiabIL3b0i\nVxfX1u/LMAwjY4TRNLsD80REXf0nVLVQRK4H7nWa4T7gelf/98CfRWS1+/0hVX3fXd8APALkAAtU\n9SW/DvC4iKwDtgGTAVS1TERuBZbiratGnEMIYAYwx91f4fowGoi2bTuxb99OAHP+GM0aUdWGnkNG\nERFt7u+xMSAicZ0/kcgo7G9hZAoRQVUzvoPCdgQZhmEkgAlNo97IyelYFb+Zl5fX0NMxjKSwrYdG\nvTF9+vyq60hkVAPOxDCSxzRNIyVefXYeMy40AWi0HEzTNFLi61dM4kBlBW2yshNq55vqubm5lJaW\nZmh2hpF+zHtupISIoH3humERep58XsLtzaNupAvznhtNimQEpmE0RUxoGoZhJIAJTcMwjAQwoWkA\nhx9LEfaV6pZJi900mhrmPTeAFHNgPpZ8yJHFbhpNDdM0DcMwEsCEpmEYRgKY0DQMw0gAE5qGYRgJ\nYELTMAwjAUxoGoZhJIAJTcMwjASwOE0DgNzc3Ko4yZycjjXiJw3DqCaUpiki60VkpYisEJHFrmyQ\niLzjl4nI0ED9gSLytoisdu3auPLBIrJKRD4SkXsC9duIyBwRWef67B24d62rv1ZEpgTK+4jIInfv\nSRGxL4AUKC0tRVVR1aoD0gzDOJyw5nklkK+qZ6rqMFd2G1CgqmcCBcDvANzplI8D16vqaXjnlh90\nbWYDU1W1P9BfRMa78qlAqar2A+4Bbnd95QI3A2cBw4ECEekcGP9O19d214dhGEZGCSs0JUrdSsAX\nYF2AEnc9DlipqqvBO4ZXVVVEjgY6quoSV+8x4DJ3PQF41F3PxZ2bDowHClW13B3dWwhc6O6NBp52\n148Cl4d8L4ZhGEkT1qRV4GURqQD+qKoPAtOAhSJyJ55QPcfV7Q8gIi8B3YC/qurvgJ7AxkCfG10Z\n7ufnAKpaISLlIpIXLHeUAD1FpCtQpqqVgb56hHwvhmEYSRNWaI5U1S9F5EigUEQ+BK4EblTVZ0Xk\nSuBhYKzrcyQwFNgHvCoiS4EdCcwrTPbl0BmaZ86cWXWdn59Pfn5+AlMx6hORw/+sdiSGEY3i4mKK\ni4vrfdxQQlNVv3Q/t4jIs8AwYIqq3ujK54rIn1z1jcDrqloGICILgMHAE8CxgW57UW3Sl7h7X7g1\n0U6qWioiJXhrosE2Raq6TUQ6i0iW0zaDfR1GUGgajZuCgsOPvohEMn6CgdEEqa0ARSKRehk37pqm\niLQTkQ7uuj3emuV7eALufFc+BljnmiwETheRHOfRPh94X1U3AeUiMkw8dWIK8JxrMx+41l1PAl4L\n9DXWCchcPE12obtX5Ori2vp9GYZhZIwwmmZ3YJ6IqKv/hKoWisj1wL1OM9wHXA+gqttF5C5gKZ6z\n6AVVfcn1dQPwCJADLAiUPwQ8LiLrgG3AZNdXmYjc6vpSIOIcQgAzgDnu/grXh5EG/JjNxhKvmZOT\nW2W2m6luNDRxhaaqfgoMilL+Ft66ZbQ2fwH+EqV8GXB6lPL9wFUx+noET9BGm9fwOidvJIUvlKKt\nLzYE06dXC0kz1Y2GxrZRGoZhJIDtojEalEhkNN7KC4QJiPBNdTPTjYbChKbRwGhUj3ksfFPdzHSj\noTDz3Mgon773Kltvn8Anq1+LX9kwmgCmaRppoaaZHYWnb/Veh2Eao9G0MKFppInoZva+35/KRyJ0\nGHUrxw+w9ABG08eEppE5tnxAzr5SBt60AbJbN/RsDCMt2JqmkRKRyGjkU4hqZi97EAZ9J2MCU0QO\ne+Xl5VXdb9u2U9Ryw0gF0zSNFFG0L8ycUlmz+OBeWPU4XLckerM0UFBQdFiZn30eYN++nVV1guWG\nkQomNI3MsOZp6DEUcvvW67A5OR2rdjLl5HSscS/8Dich6NRSDR8SZTR/TGgamWHZH2HETfU+bF17\n5fXuglB9yLQIBUWeoIyMMu++URNb0zTSz5YPoHQd9P9GQ8/EMNKOaZpG+smgAyiVnUAyLYIAlVE0\nTpl2C9G2c+Z0tG2bRk1MaBrppYEcQGHbxHYIaZVJHmT6fLdt08x0w2HmuZFeGsgBZBj1hQlNI70s\newCGXN/Qs0g7vpleV1yo0TIw89xIH1s+gNJ/NUsHkG+m18bM9paHaZpG+sjwDiDDaAyYpmmkjHwK\nRLIoaJuXUQdQJhGCWqNQUFRZV/UqfLMd7PyilkIoTVNE1ovIShFZISKLXdkgEXnHLxORobXa9BaR\nnSLyk0DZYBFZJSIficg9gfI2IjJHRNa5PnsH7l3r6q8VkSmB8j4issjde9KdfGnUMwUFRWhfAG3S\nDqDKuwtQ96ozxV0tps8vpaDI87yXlZXVuLdv36E0z9JoDIQ1zyuBfFU9U1WHubLbgAJVPRMoAH5X\nq82dwIJaZbOBqaraH+gvIuNd+VSgVFX7AfcAtwO4Y3tvBs7CO0StQEQ6B8a/0/W13fVhNCTN0AGU\nCkOG3MugQdcxb94LDT0VI42E1c6EwwVsJeALsC5ASVVlkQnAJ8DuQNnRQEdV9e23x4DL8M4xn4An\neAHmAve56/FAoaqWuz4KgQuBvwKjgWtcvUeBmcADId+PkUa2HHAXURxAtYPREznaoqkRNNUDpcya\ntZfLL7+4QeZkpJ+wQlOBl0WkAvijqj4ITAMWisideEL1HAARaQ/8HBgL/CzQR09gY+D3ja7Mv/c5\ngKpWiEi5iOQFyx0lQE8R6QqUqWploK8eId+LkWaW7XQXMRxARQXe9+GoSKSeZtQwBD3sT9/6NNmt\nt7Ny4Xf58Y+vqaOV0dQIKzRHquqXInIkUCgiHwJXAjeq6rMiciXwMJ6gnAncrap7Ujg3O0zD0J3P\nnDmz6jo/P5/8/PzEZ9RCSSSlWpgtjofXEQoKQjhdVDmj6NsAlHz4Oj1PPi/0vBKlplOoujSsc6h8\nczn/WvIvbvzLjaxc+F3TMjNEcXExxcXF9T5uKKGpql+6n1tE5FlgGDBFVW905XNF5E+u+nDgChG5\nHcgFKkRkH/AMcGyg215Um/Ql7t4XIpINdFLVUhEpAfJrtSlS1W0i0llEspy2GezrMIJC00gMX0uM\nxfpFf6HV6+vYOukqBvUdkHBfobVPEU4oeRGAHSvmZVRoRt+bHl5LXvzMYs4YfwY5HXLSOS2jFrUV\noEg9WTJxHUEi0k5EOrjr9sA44D08AXe+Kx8DrANQ1fNU9XhVPR7PqfM/qnq/qm4CykVkmHgq6BTg\nOTfMfOBadz0J8I8uXAiMdQIyF0+TXejuFbm6uLZ+X0Y9surgbi7uCrt2vp7xsd46wfP1dTqz8Z41\ndGDvAVa8uILhE4c39FSMDBFG0+wOzBMRdfWfUNVCEbkeuNdphvuAMK7TG4BHgBxggaq+5MofAh4X\nkXXANmAygKqWicitwFK8ddWIqm53bWYAc9z9Fa4PIw10atuWnfv20TEnvqbUu9fXmH/cq3x3wAE+\n3K0kc7pkIpmLBBh4TB+2JTjGLZFRJDe7xFjx4gr6ntmX3GNyMzyS0VDEFZqq+ikwKEr5W8DQw1vU\nqBOp9fsy4PQo9fYDV8Xo4xE8QRttXvZ1ngF27tsX1yz38Uzyk2l7xJ85pnI5X+4dkvB48TzqkYhU\nZSm6YNkfyVr3AoVDv5/QGEr4JMTJUllRybtPv8vlv2i8mrCROraNsgmgCpfM/pyvTYkw74VXG3o6\nURA+2vF1+nZ4jdayO371FFjW/2LO+KSQ7IoD8SvXM2vfXku7zu3odWqvhp6KkUFsF00D0CmvEzvL\ndsavGIW3Hvd+NrYte7sPHc1X+07n+I6vsHbHhLj1R0X8pL+JGcxlHXuyKfcEBmx4g9V9xyQ32QSo\nTk4cf56L5i7i7Eln14jVtG2WzQ8Tmg3AzrKdvKKvJNRmzl+/4rH3xjF9xOOcf8lALpALMjS75Fm/\naxTDuv2eTq03sONg7zi1NelA92X9v8FZHz5bL0IzVnLi2pR8WEL5V+UMOK9mBEEwdtMyIjUPzDxv\nIky+ujvH92lNqxNHN/RUYlKhR/CvnePo3+kFhHAxjcmwttc5HFn+GV3LN2RsjERZ9NQihk8cTla2\n/Us1d+wvXI90yuuEiNAxt2P8ylG4fNjnPLvYC3XtmNsxdCLcdnnRE+jGeqXCln2ncqCyAz3bvZtS\nP5GIVL1qU5Hdmn+ecCFD1jWOPd1+MPuZXz+zznrBRMaWvLjpYuZ5PZKMWR7kvFM388DL/fh0c3vm\nlc6rKo9nqu8t285tVZFa8ZkuXZKeIwjrdnydwV0fYvO+UzlQ2Snpnuo6C2hZ/4v57os/5LUzp3Io\nu03SY6SDsMHsZqo3D0xoNiFaZyuXDCnhucW9uOmStVXlvtYJgIjnbg/QNjcVIZg4eyu68sWeoZzY\nsZAPyq9MsheJu4XzknOGcHIGHEKxTqaMhh/Mft0frkvrHIzGiwnNJsYlQ0v4zu9HMHXMx3Rs6+Vr\nrK11JqJVZorPdp3LWd3uJ7fNx5QdOCHh9vH2o0cikkGHUDjnD1gwe0vE1jSbGHkdDjC831ZeXNG4\nkzpV0pp/7byIfp0WIGQmGW9DO4T8YPYRk0Y0yPhGw2BCswly+fCNzF/Si4rMOajTwrb9/dlz6Eh6\nt387I/0n6hCSaRGyEki8EQ8/mP3YU4+NXxmYNTGPyCghMkrIzTXNtKliQrMJcnLPHXRpf4BFH3Vr\n6KnEZd2OC+nVfhE52WXxKyeBv0OoVZwdQgUFRd7RHGkc2w9mD0tZWRmqiqpakHsTxoRmE+Xy4dXh\nR42Z/ZVd+Hz32fTr+CKJnL0TFn+H0Mkb3kh733URK5jdaP6Y0KxHcnNzuUAu4PK81BM6nHfKZj7b\n0p5PN7dPw8wyy+e7zyGnVRldj1hbo9yLw0z9EVzW/xsM/ej5lPtJBAtmb7nYX7weKS0tRVWT3nce\nJBh+1NhRslm34+v06/QSWeKZ0UUFBS6TUura59pe59CtfEO9OYTKN5fz8dKP4wazG80TE5pNmEuG\nllD8fnd27m38kWPbD/Sl/EBvjmuf/mTF9b1DaPEzixk4bqBlZm+hmNBswmQs/EiEUZFI2g9C+3jn\nWI5pt4J22VvS2i/EdwjdEhlFJDIq5STEqWZmT9d2VaPhMKHZxMlE+NFtlWUZCZA/UNmRz3adR79O\nC0i3UyieQ8hPQhzt/J9ESDWY/TZNbEur0fgwodnEaUrhRwBf7DmLVln7OCrnvbT3nWmHkAWzG2Db\nKJsFfvjRyJO3ggQSbohwW2UK8ZHOTPcJewRGXShZrNtxMUN/9RDKPJI/tSf63nQB7h2xgW2d4+Xz\nTJy1b6+lfZf2oYPZASKjs6pzAZhJ3iwIJTRFZD1QDlQCB1V1mIgMAmbjHZJ2EPiBqi4VkQuA3wKt\ngQPAz1W1yPUzmJoHq93kytsAjwFDgK3A1aq6wd27FvglnoX1a1V9zJX3AeYAecAy4N9UNTP79Ro5\n552ymQcK+7F+c3teqazOopRqouKgwE0t81FNdhzshQKL7vgRL+6clVQfsfamRyLCkHV/p3DoD1KY\nYXQWPbWIEVcmqGWqmjnezAiraVYC+aoaVFtuAwrcyZQXAb8DRgFbgEtUdZOInIp35K4fFzMbmKqq\nS0RkgYiMV9WFwFSgVFX7icjVwO3AZHds783AYDwlYpmIPKeq5W78O1X1KRGZ7fp4IPmPounihx89\nWyv7UZXWmarGmSFG/PQ+4L5AicRN1BGGMz4u5LUzv3tYyjiZFkGIfq55jXoEU7cJBUWVXjD7Zgtm\nN8KvaUqUupVAZ3fdBSgBUNWV7oxzVPV9IEdEWovI0UBHVV3i2jwGXOauJwCPuuu5gJ+efDxQqKrl\n7ujeQuBCd2808LS7fhRo0UcARgs/eqXyFS9/p6Z/J06qFBUUsOa3l1J+b0+KCm5OW8wmwKa8Ew9z\nCCWyjbLy7gLUvfw5WTC74RP2CVDgZRFZIiJ+4sBpwB0isgFPM/xF7UYiciWwXFUPAj2BjYHbG10Z\n7ufnAKpaAZSLSF6w3FEC9BSRrkCZqlYG+mrcaX8yTF6HA4zon5nsR0sf8xJuvL6mJK39bto7CEU4\npu3ytPabboeQBbMbQcKa5yNV9UsRORIoFJEPgSuBG1X1WSccHwbG+g2caf6bYFkChFkxD72qPnPm\nzKrr/Px88vPzE59RGvG3U4KXQDiYDzMVLhu2kf+eexpXjNhAuhSi3duE524YCEDB3/4UtU7yDiLh\nox0Xc0bu42zddzJA1OMtEjPbhasfmeldLhpVlf096DQSl+kojKkOLjP7uPiZ2X3COH/a5naxUypT\npLi4mOLi4nofN5TQVNUv3c8tIvIsMAyYoqo3uvK5IvKQX19EegHP4Dln1rviEiDoduzlyoL3vhCR\nbKCTqpaKSAmQX6tNkapuE5HOIpLltM1gX4cRFJqNgeA/SDqDnIPhRyNP3ppyf7u3CX+6oD0nXvAV\nHzwLV0VeYcB5NRP+pnpsQ/Do31jCN5Ege1+4XrBsBuf+/bYa92r3H7bfFS+u4PoHrg89hzDOn5ml\n66uu0+lka0nUVoAiad6MEYu4+oiItBORDu66PTAOeA9PwJ3vyscAH7nrLsDfgemqusjvx61zlovI\nMPEkxRTgOXd7PnCtu54EvOauFwJjnYDMxdNaF7p7Ra4urq3fV4smXdmPfIHZb9whpjxzFMBhAjNd\nrN81irwjPqZT6/TtHV/W31tFipcyLhaaVcGyE14CoHvfNnQ52gSb4RFG0+wOzBMRdfWfcB7z64F7\nnWa4D/DXOm8ATgBuFhF/JX2cqm519x6hOuToJdfmIeBxEVkHbAMmA6hqmYjcCix1/UScQwhgBjDH\n3V/h+kia3Lx2bC/bW7MshNnkt0vexJLooUFCjfChsATDj/octTuJ+dQUmBf9dl+c8MLqWM5kzfTg\n0b/Ltv0HShajIuHP6YlGWUfviI3/+u/xddaTQFJidaa6ZlWw7eJCyrZ6S/C9+7wO3JTwHMLim+pm\npjcN4gpNVf0UGBSl/C1gaJTyXwO/jtHXMuD0KOX7gatitHkET9BGm1dyG4CjsL1sL9u1pjnXRaaH\nbhembnSin0eTrNkbM/woJIkJTCgo8szhVM30LftO5Zi2K+jZ7l027jkbUAoKkvSmV1bAB08B8Hxf\nKBnZjZNOuCFqVV3sCUoZ5glPX2ACvPHrC4GHOTH/35KbR0h8U93M9KaBxU80Q5LNfpSowEwvwrod\nF3Fchzdok7UjuS4qK2D1HJh9Orx7LwCrz+zOpMH7OaXzXFpL3Zp3UGB2fWEcr632ljmOHVR/0Wzz\n5r3KqFEzmDfv1Xob00gM20ZZB11y28Z11HTJbRulrhA25jCnY6zEDwGzPUFT3Q8/emlFDyadE26d\nMJrAnC65pGIiJ8reim5VR/8mhK9Z/uMWyOkM4++GE8bBLVmMOP17rC4/SJ+ORQzt9gfW7biIrftP\nidqNLzCPPHEJsNiV1t83R9vcLkyc6P3Ni4urrR4z2xsXJjTrYH3pzKTqdpHpoY+AjYVv9kJypq8f\nfjRxRHyhGVvDDH+Ubbrwj/4NRSxhWeuLrpLWfLJzHFv3DeDkzs9xVM4HrNtxUdV9lYqq664vjAMW\nc9roufQ4qQdnXxX+DKBUmVm6nvfm/YP/nXgp335mPqdffj5gZntjw8zzZkrY7EcNa5Ifjn/0L0A2\n+2NUqmWGj78bvvM2nDi+zqQYOw4ey9Kt/8H+yk4M7fYHwBOY206oNsmlMhugwYLZfUHp/zQaH6Zp\npokuMoPMmbLOVE/QTI8XfhRLYFab5Q0jQbft748AN0f8YHIX3B5Cs7wlInUujORH/qfGX6nsq/tY\nkeXtBs668b+r6iUSzB6NoHaYasIOC4RvXJjQTBuZM2WT9VCfd8pm/ljYL+q9ujXM+jfLa/NW5EaG\ndvsjy7Zdx9k3z+K935/CyQd20rpTr5hmOFQnG4aa4UQ17juPeWXlPpZWPMubgybwze+/yRP/PpZd\ne6BjfoThV6QWmOFHYiQfVVGNBcI3LkxopgmRwzPjpItbRlVrT4k4h1pnKxcPKeExaubYvHnL9gZ3\n+sSj5tG/8MzWNd6NHRvhiQtjtpNa19F2/fjhRdXcDsAT/z6WP7st68kEs98yWlDNfNrMupyTpolm\nnhYvNBXl/65+F4C/v7qeS8b0SaqfskpPs3jw95/ysx/+gTWvv8CA8y5O0xyrtSOfw//xo3PJUE9o\n/rxQadvaE+yNxekTj893n0N3t/b4rRN6c815I7h4YPjUbNH2lcu0SI14XEWZwx6+L7dQUQH3zEl+\nvqocFuubCeoaIx2arVE3Ld4RVAn8tc1BAB56+auU+3v0wS8AWPTMkyn3lQ7yOnjbCFdsgj3lXllj\ncfrEwz/6F+B/f/ithARmWAThHI4A4NnXlaOSO/rHaEG0OE0z2nZJAOnSiUHfHgXsSan/n998FFOu\ngONO/3pK/QRN8kRk2wUylsNjRIV3P1dWPSKA8I/bc/jH7TTa5MRBth/o6120eh0OpZaJPha9yQaB\nK6ffUlWWTJiX/yXUJWtGWo+4mJHVJXR3sWKLzWxPHy1OaEbbLgnwIV2YwigmUcQpJO/tvHSi53gp\nWVMRp2bdRDPJw7aMZmbf8uRedulazrrsz1xw/TW0adsm5a2P9Ur2Cqg4A/TItHctCD997pf84Udt\nuOOhMiaPOSa1DtN8xEUiZn+s2GIz29NHizfPfU5mOzezjBs4lzLaxG8Qhy2fbeGrj1M399PF0e/O\noPJfz7H6tdXcNekunvn1M4B3wmKT4NB50Cr9R//6LHtrDR1WXsntR4/N0AhGc6HFaZp1cQkbeJ88\nbmIkD1FMqxT+fc6acBaL5i5iwvQJaZxhNUKsg9Oia4/nThzHu/Ikwy+/ht4D81n92mreewV+0Sqv\nznaJzChCePFBAAAgAElEQVTdJ1fW6P3HL7lRbkn57PJoTB3bnQdvn8nun/+EBxnA9axJ+xgNSTSz\n3Uz25DChWYufspKpnM8dnMEM/pl0P0O+MYT7vn0fY64bQ4e8DmmcoUdlwHSXYZG4nu8B511cw5s/\nfOJwhk9Utm7YysrClbz3ynvMnjqbgWMHcvqY0+l0ZKeE5pPqts+68AVwp9YbGfKrh/AyESYfeB6N\nS8b04ZIxffiCpVzBOM5gK8PZktYxGpJoZruZ7Mlh5nktslHu4W0K6cV8jku6n3ad23Ha6NNY8uyS\n+JUbkG69uzHmu2O48S83ctGPLmLb59uYPXU2j//0cVYuXMmBvckl8c0EOw66Q01bFWVsjB7s4XYW\n8Z+cw+Y0C2ajedDiNM3c3Nyqb9guuW2jfwNzgPt5kymM4kTKk3YMjbhyBA//+GG+9q2v0fqI1qlM\nu06iHTmbcB9ZQp9BfegzqA8X/fgiPnr7I1a9vIoX73uR/mf3Z+DYgRw/5PiQpzFm2FT/0WKExUmZ\n6UK1hiVSHV8b5Fw2cTUfcxMjeYzXEl6mEam5maCxRiiEyeLlY6Z8NS1OaIY9nyfoGHqGheSSuMbV\n9diu9BrQi1WFqxjyjSFJzTcMtU31VGl9RGtOHXUqp446ld3bd7P6tdUUPVzEc7c/x2mjT+OMcWdw\n9IlHx2yfaVP96LYrGDBjPsnsjw8K2mjbLH1uYDUr6MadnMH0BJdpgoK4MZvAiWTxaszvo74x87wO\nLmEDF/I5NzGSQ0k6SkZMGsGiuYvQyqbpk23fpT3DJw7nuj9cx7V3XUurNq2Y86s5zJ46m7fmvMWO\nLUkmDE6BTXvdQQLZ6T36N0gWcAdvs4DevFx10rRhtEBNM0jQVPepbbKn6hjqM6gPrdq04l9L/kW/\n4dGTZ6STdJjqsfDXP0d/ZzSfrfqMVS+vYvbU2fTo34OBYwcy4LwBtGlbO1xLYp76mMrRvwDy478D\nfw99FG+i5HGAe3mL/+A8+vMyx7Er7WM0V9rl5bK3rOayVnMx8UMJTRFZD5Tj7To8qKrDRGQQMBvP\njXkQ+IGqLnX1fwF8BziEdzZ6oSsfTM2D1W5y5W2Ax4AhwFbgalXd4O5dC/wSzxb7tao+5sr7AHOA\nPGAZ3nHBhxJ589H+gLVNdt8xNJFxnEIZl/JZIkMgIp62+dSiehGa6TbVo5HI+mcsoZ2q2V5UUMAJ\nHV+ileynx38mH+UQj0Fs44es5od8jad4mRxS27TQUthbtv2wAP/mkqEprKZZCeSranBF+zagwJ1M\neRHwO2CUiJyCd0jaALzzyF8RkX6qqnhCdqqqLhGRBSIyXlUXAlOBUlXtJyJX46WdmeyO7b0ZGIyn\nXiwTkedUtdyNf6eqPiUis10fD6T2cUQnVcfQaaNO49UHX+Wrj7+i+wndMzHFBiP59c/YGmg8fA11\n/a5RDOv2e6+3OtYnvdEO10bDOIUAvs06lnEkEYbwm6pjMFoeYZ1G4OUAjVbWHPKChhWawuHrn5VA\nZ3fdBShx15cCc5zWt94dyztMRD4DOqqqH4PzGHAZ3jnmEwD/iZ4L3OeuxwOFTkgiIoXAhcBfgdHA\nNa7eo8BMMiQ0ITXHUHbr7IwHuzcG/PXP4ROHV8V/zvnVHI5of8Rh8Z/JLhsENVT/6N+ds96oOvo3\nFtEEdFinkAC/ZjFXMI6nOJ5JfJLU3Js6qW4NbS55QcM6ghR4WUSWiIh/vvk04A4R2YCnGf7ClfcE\nPg+0LXFlPYGNgfKNrqxGG1WtAMpFJC9WXyLSFShT1cpAXz1CvpekScUxNOQbQ/jwzQ/ZVdoy1sXq\nK/5zy75TOVDZgZ7t3k1Lf7FozyH+H2/yO87gA5ruP7yROmE1zZGq+qWIHAkUisiHwJV465XPisiV\nwMPA2DTNK4xECi21Zs6cWXWdn59Pfn5+zLrx4jjDOIZqJiSuyZJnhzHqO6PCTr3JU9f650nnnMTA\nsQPpO7hvyPjPqCOwbsdFDO76MJv3ncqBysR2MiXCiezgZpbxI77GPBbSiYMZG6s58tu8vpSVeSt8\nHTp0TLm/4uJiiouLU+4nUUIJTVX90v3cIiLPAsOAKap6oyufKyJ/ctVLgODBNL1cWazyYJsvRCQb\n6KSqpSJSAuTXalOkqttEpLOIZDltM9jXYQSFZjzixXGGcQzFWhvrItNZ+vzSjAe7N1airX++9tBr\n7LhtB6ePOZ2BYwfWGf8Zi+DRvx+UX5mBmVdzCRtYzpFMZwT380YjyHHfNNizVygrK6P1/93JwVkd\nGDp0Rsp91laAIkmukSdK3K93EWknIh3cdXtgHPAenoA735WPAda5JvPxnDhtRKQvcCKwWFU34Znd\nw8STRlOA5wJtrnXXk4DX3PVCYKwTkLl4muxCd6/I1cW19fvKOL5j6L8ZnLCp1usUL9i9pVM7/jO7\ndXZK8Z+f7TqXjq1LyG3zcYZmXM0MVrCFHB4k/UmRmxt79goLXzuC393n5V+YNXAl+UNn8OMfp8so\nrX/CaJrdgXkioq7+E85jfj1wr9MM9wHXA6jqByLyN+ADqkOR/MjuG6gZcvSSK38IeNw5jbYBk11f\nZSJyK7AUb101olq1Gj0DmOPur3B91BvJOoZGXDmCBfcuYPDFg5GsuvQUqQoZEmqGEjU3Eov/rPa6\nB+M8/aN/+3VawJKt30cTDEEO60kHaEMls3grVGKPprKlMh6Jvo89e4U33mlD0SU90L1e3dzcXL53\n9Ui+d/XITE83o8R9slT1U2BQlPK3gKEx2vwG+E2U8mXA6VHK9+OFKUXr6xE8QRttXqkdGZgiyaSS\n6zOoD61axw92z+RWxMZKmPXP/3rlEFnZWVE/k237+3NM2+X0bv82n+0+L6Gxw3rSfYKJPZ5hIUex\nL2q9prKlMh5h34cvLBctbcOpJx9C95ZRrTM1D1r0jqB0kOiOofoOdm+qRF3/fLh6/TMW63ZcyNBu\nf+SrfaezryKzB/6kmtijOVFbWP7oul3k5SqNO8dXcpjQrINo2yx9/OMHktkxlHiwu2eqJ2qmV2+p\nTO92yvqmdvznqpe9NeGps2HsQBhzOvjpP4NH/763/RoyfRyxn9jjLgbyc1bWWbe5mOo+FZVQ9MRy\n3lzciYrup3HGQKkSlr6nPDe3+Z1UZwk76qC0tBRVrTIvXtFXeEUPP2s8UcdQdutszrrMC3YPQ0FR\nJQVFmrAeU7m4wJ0z1Hw0oG69uzF66mgAfnQRfL7NE54/fRwWroS9B7yjf3NaldH1iLUZn4+f2OMF\njuOVOIk9yipvY7t6L5qgyXqoUlhe4kU3/Pkv7Yjc3oniVSey92APjnrrZ1x56V7ycr33VVbmmeVN\ndddPXZimWQed8jqxs2wnAB1zA3FlcrjTIFHH0NBvDGXWt2cxpjQzmd2DZDKJR8MhTHu0eu1x2Sfe\n67fPBuv8NVA7kZ7DO4WgZmKPfmlK7JHICZSZ4lClsOrL7ry5vjc3f200BM5HHTroAJMm7OXuo0/h\n4K5dbASmy31VbZujhuljQrMOdpbtjKpZvlJZXRY8pycRx1DbTm05bZSX2T3Twe71kcSjvokl+P31\nz1Uvr+KLD6/nF986j1HduzP22FNC952oUwjSn9gjkRMo00VQSL752bEs2tCLXp13MPK4DYDyX7vK\n6dDef6a93Di7du1qdo6eeJh5nmZ+ykoE5Q7OiFt3xJUjWPr8Ug7ut50l6aIq/nO2t9t3+MStdLvp\ndQ4ctTnjY3+bdZzADiJkLuF0OvHN7VlvDeOqv1zB8bf/mB/Ov4jPyzvyrUHvsfxHf+Tt7/+Z3339\nVYCAwGzZmKaZZhJxDHU9tmtVsHu4zO7pid3MTAhT4zD7bxklNfT74qf3c/Xuk9ky8QU6rD6FTouH\nIhXZGRk71cQemTbJ69IkvzXoPe6fsIBu7ffWaNMla4an9gYmFNwO2ZzN8FiY0MwAiaSSCx/snr7Y\nTc1Awt6wZmymUaK8v21Q8cRZlI75B19d8xR5haNps/mojIzvJ/b4JmM4ldKE0gj6JnkX+Tlllben\nPJdkhGS0SdXObuQ7eVoqJjQzxMlsp8A5hp5mIXkxHENhg92N1Mje055uz1/EnpPWsWVCZrXOhkrs\nkRYhacTFhGaqOE96NC/rxWxgNXlMc46h6uxH1aZs8sHuEtWp01BbLmt66NPfe23Tv7YZHpxH7F6E\n9mv7k/N5z9Bap+9JD+NFD1IfiT0yJSSrTHKoMstbukkexIRmivie9KAXPUhwx5D/T1c7YD6ZzO6Z\nOkYiWTJxRo9PNNM/qhkekkS0Tv99JbP8MIMVfJMxPMgArmdNUnMNUm+apJnkdWJCM8OEcQwFg92b\nc2b3xkQyWmeiJJLYIxpmbjdOTGjWA7UdQzWTFHumZ30Guzc1Ypn+0bS/RE+mzPRaZ12JPWof+SAC\nBS93hHPg/Flf59M9/RpMSNaeW0s3yYOY0Kwngo6hjyvvrnIM+aZ6OoLdlYYP+ckEiQjBZMzoTGud\nsRJ71A5eX043vvHSN6AzHNr0Ist/WdRgmqSZ4rGx4PZ65GJ3xtC0GGcMpRrsXn6Ut43thWWZT8Tb\nHPG1zo5LBrNlwguUn/0ump2eI3t/wPscQQV3MTDq/eV04/ucy03dH2XUwCv4xZhPzfRupJimWc/U\nlUou8WD3mqz+y3IAnu+4l4vTMtumh1CtbSZqqnttomudqZKNcidvczkXMpitQJS8lAI/q7wNzqz/\nNLF9835r3vGQmKZZz/iOoUJ6MZ/jDrt/9qSzWfT0IrQycfPomG5eHucel57Ikm7N01SPR+XdBaj/\nSqGf2longLr91sniJ/b4JcMAL2vW/fomJ+gn3KFvN2gyKt873lwzE6UTae5rFyKiyb5HEYmasCMa\nF2RdUPXQh4np+5AuTGEUH2X1j/HPkty2xMgoYevKm3n4pAryv8jirK01vxezhkUCuWoyGyrU0GRN\ni9Qph8K+/4p2u2n9vTuiZnOL9rfOlenx5Z8I91a+ySyO4eOs40HDPTfppLZ22dSFpYigqhmPuTPz\nPE3EynwUC98xdEflB4ftGJrz+CG+N+WXSc+l637hO2uzefgkbz0uKDibY8ajWMQTiGGdRtl72lNx\n183sOWkd2897q4aHPWYMqfucZVik6otXUfayl3J2MGvPcfzhUBvOW/42HzdARiOw2MtkCWWei8h6\nEVkpIitEZLErmyMiy93rUxFZ7spbicgjIrJKRN4XkRmBfga78o9E5J5AeRvX3zoReUdEegfuXevq\nrxWRKYHyPiKyyN17UkSa3BdALMfQxKu9cJevPv4q6b59wVnco7LFmurpxF/rPPqJqzjQbRtfXfNU\nnZmTKlG25HgCaR3rWM5y3uANVrKSbWzlmIXPs/PdXJbP+kd9vQUjTYQyz0XkE2CIqkbNzy8idwDb\nVfW/ReQa4Buq+k0RaYt3KuX5qrpBRN4FfqiqS0RkAXCvqi4Uke8Dp6vqD0TkauByVZ3sju1dCgzG\ns6aWAYNVtVxE/grMVdWnRGQ28E9VfSDK3OrFPA+SiKlegTCV8zmZ7TUcQzWdBHWb6pFRWUSz8QXY\nYqZ6TOKZ70GCn4+iVVpnrw4bola+5eAvaXcQftru13HXKhvKLG8OJnmQxmaeC3VrpVcBfnChAu3d\n0b7tgP3ADhE5Guioqv5ZS48Bl+GdYz4B8P9j5wJ+CujxQKGqlgOISCFwIV5K7tHANa7eo8BM4DCh\n2RAkYqrH2jH06bbfcuaJ+3j3gxxOOmZGnX2AUlBw+H9mJCJmqtdBmC8JRdEj9pP9g9vYe9wGKjrs\noqLDbio67KL1tjxgAx9WHsvfdl7DiYfa0GO30HMP9FgptKsQ/vPd6GPIsEiDmORgZnmqhBWaCrws\nIhXAH1X1Qf+GiJwLbFJVPzhwLp4Q/BJoC0xT1e0iMgTYGOhzI1QdqtIT+BxAVStEpFxE8oLljhKg\np4h0BcpUtTLQV4+Q76XRES2VXG6ecMXkbP50f2oeW6h7jbMlo1JJRbs9TgjuriEQq8t2Q2UW/AB2\nDv4n2bs60GpXe9psPpK2n/QF/snf5RI+nPEm//XvYxv6LRn1QFihOVJVvxSRI/GE5xpVfdPduwZ4\nMlB3GF4u/KOBrsAbIpKojRtGxQ6ths+cObPqOj8/n/z8fKDmGUDRqHEuULJEOU8oGrVTyR2f9ZPD\nDt+KZYbX9VH4GqQuLmhRgrMy+1AUAVjrut1esvYdQfauDmTvak/2rva02tWB1huOrVGWdbANQoTu\nfVZEHWv5D1byzYHHx51T1vBIg5370xzjMIuLiykuLq73cUMJTVX90v3cIiLz8ATjm84En4i35ujz\nTeAlpwVuEZG3gKHAm8CxgXq98DRH3M9jgS9cn51UtVRESoD8Wm2KVHWbiHQWkSw3TrCvwwgKzSCx\nzgBKJ4mY6sFUcn6mmUcunc+a533/V3QzvC6KCgoYFfEEZ3PQOH1z+VCd2uEuKlsfJHt3+yrh12pX\ne1rt6MQRXxzjhGEHsne3QyrD7TGPZcrLtAhPhNQwG+LcH5/maJIHFSCASKR+lpjiCk0RaQdkqeou\nEWkPjAP82Y0F1qjqF4EmG/DWG59w9UcAd6nqJmd2DwOWAFOAWa7NfOBa4F1gEvCaK18I/FpEOuOt\nqY4F/AW+Ilf3r67tc4m++caIv2PI59yfnMma50kq2D0ajVlwJmIu+1pg0FzO/qRvtXa4ty2S4TPP\njZZJGE2zOzBPRNTVf0JVC929q6lpmgP8HviziKx2vz+kqu+76xuAR4AcYIGqvuTXAR4XkXXANmAy\ngKqWiciteB50BSKqVYn+ZgBz3P0Vro/GjUTZOhezrnDXXUVc0Mf7x79lTGrCrfbZQvUtONNtLjdm\ngmZ4EDPJmwdxhaaqfgoMinHv36OU7cbzpkervww4PUr5/jraPIInaKPNq/436aZA0FT3uUAuqEr4\nWr5DWL2mNas+aE3Jrj3s6lHBE1f4D7vyX5INvz8FegyBY4ZAj6Fw9CBoU3cquaKCatMylqmeLA1l\nLjdmGtIMD9IcTfLGQJMLCG+OvPVuG1Z90JqvNmcx4KRD5I/cz/73Xufd/1fMOc+cx/9OhJHHjWL9\nWXM4vms/+GIZfLkMVj8Jm1dD5+OqBekxQ+CYM+MKUogvOM1cNozDMaHZAGzdcQRvrDmSf3zQHRDm\nj2hXdW+5eyHCbZXVewku+D9F1R0cPQiY6l1XHIQt70cXpMCSv95L62HjGNR3QFXzWDGZf7rgH5xd\nsZluR2U1K3O5JWEmeeYxoVlPBAXlZ5vbM+KkrUweuZ7BB1+hTavDTagw+9cByG7tCdHDBOkH8MAg\nrj5jO73P/ltVdZ0dvRv5Poz4YhOvf7iH73S/sFmZyy0JM8kzT4sWmrm5uVXCqWNuR+aVzktr/zEF\n5fGlUQVlIkQi4Uzd/9nUm5PfGVFD04zRIw89eoiLjjybIyqPSWluLRXfAWQOn+ZNixaawX23kqYn\nPZOCsjYFBUUx713+5v8w6NWXmZx/mK8uJnef9a10TKvF0lAOINMu65cWLTTTRX0KyjB02FtK/43v\n1Pu4htESMKGZJPUhKKtPBExMCx7y0fO8f1w+rP17VYhRPBqbXzvvlllVJmey5LbLofTXIeNiEyDr\nJzXjMKtiYBvILDeTvH4xoZkA9a1RJrplEiC74iBDP3qexy/4HQVn/ycAkcgotImlfUuHyZmuJZfa\nWBxmy8aEZhwam+kdjwEbXmdr595szu3b0FMxjGaJCc0oNFZBGcZjPnzNM7x12jVx6zVGgiZ5OkzO\n3NzcpM5Bj4eIhN8OmwBdctuyvnRmnXXMU97wmNAMMO/dXo1OUNamLo95j60f0nHvNtb2OrseZ5Q+\n0m1uNrWs5GGWE8wkb3havNBUhTPv2QbAm0sOMXlc4xOUYRn+4TMsPukyNMsC0g0jU7R4oSkC2atd\noqZF9zLih1dHrTcpb1Job24igfIXyFgSTSwcDT/M6KWzfphQu/oknke8pZububm5cc3+lv4ZNQZa\nvNAE+NXEAUx8GC675uSYdRIxixLz2iaeWDgafpjR3iM6pdxXpjDTsm6a2nJCS6XxZKBtQC6/eAwA\n5158RgPPJDn8MKPFJ09s6KkYRrPHNE1HcB96rPvJENv89omvlcbzmjfmMKN0e8QNo6ExoenInGmU\nHvO7Lq95Yw4zMpPcaG6Y0MwA8bTWMITNYhQ2zMiPVxTCnfedCqZdGs0ZE5oZIF3Zk+rSLn2Gf/gM\ni0++vM4wo2A/kciopOcTFtMujeZMKEeQiKwXkZUiskJEFruyOSKy3L0+FZHlgfoDReRtEVnt2rVx\n5YNFZJWIfCQi9wTqt3H9rRORd0Skd+Deta7+WhGZEijvIyKL3L0nRaTFfQH4YUYrTryooadiGC2G\nsIKmEshX1aogO1Wd7F+LyB3AdnedDTwOfEtVV4tILnDQVZ0NTFXVJSKyQETGq+pCvJTjparaT0Su\nBm4HJru2N+Odqy7AMhF5TlXLgduAO1X1KRGZ7fp4INkPIlPk5ubWaWrn5OQyfbqnmYY1yX2SDTNK\np6keLfbSTHKjORNWaAp1a6VXAb7dNw5YqaqrwTuGF0BEjgY6quoSV+8x4DK8s80nAP5/71zgPnc9\nHih0QhIRKQQuxDvrfDTgez8eBWbSCIVmPAdTbfM9jEkONbMZJUK6TXUzxY2WRtg4TQVeFpElInJd\n8IaInAtsUtWPXVF/V/6SiCwVkZ+58p7AxkDTja7Mv/c5gKpWAOUikhcsd5QAPUWkK1CmqpWBvnqE\nfC/NggEbXmdLIw0zMozmTFhNc6SqfikiR+IJzzWq+qa7dw3wZK0+RwJDgX3AqyKyFNiRwLzC2Kmh\nbdmZM2dWXefn55Ofn5/AVDJPomY5pC/MqD696oaRToqLiykuLq73cUMJTVX90v3cIiLzgGHAm279\nciLemqPPRuD1gFm+wN1/Ajg2UK8XnuaI+3ks8IXrs5OqlopICZBfq02Rqm4Tkc4ikuW0zWBfhxEU\nmo2RsCa5T7qyGdW3V90w0kltBSgS8pSCVIlrnotIOxHp4K7b461Zrna3xwJrVPWLQJOFwOkikuM8\n2ucD76vqJjyze5h4C3lTgOdcm/nAte56EvBaoK+xTkDmuvEWuntFri6urd9XsydMmJFhGJkhjKbZ\nHZgnIurqP6Gqhe7e1dQ0zVHV7SJyF7AUz+v+gqq+5G7fADwC5AALAuUPAY+LyDpgGzDZ9VUmIre6\nvhSIqOp212YGMMfdX+H6aPZkMpuRmeqGER9p7p5PEdHG/B5FJCHz/PyVj9Jxz1b+7s7/yQSJnCkk\n0yLmPTcaBSKCqmb8eLsWFxDelEk2zCjd2DZJoyVjQrMJ0VjCjCw202jJWD7NJsTwNc9YzkzDaGBM\n02wiZPrQtFsio6qyfmZ8UcgwmjAmNJsImQ4zUgjt/DGMloyZ500AP8xo+Ylfb+ipGEaLxzTNJoCf\nzWjfER0bbA7mMTcMD9M0GzmN5dA032OuqnZqotGiMaHZyGksYUaGYXiYed7AeEmKD0+WkZPTkenT\n5zfqQ9MMoyViQrOBiWXqikjGw4wMw0gcM88bMZbNyDAaH6ZpNmL6b3yHF8/6Ub2N52c5ioZ5zA3D\nwzTNRsxbh47g409X1Nt4vnc82ss85obhYanhGildsoXyytj3fUdRKsyaNbFG7KUJRqMpU1+p4UzT\nbKQ8/dRTTB+fzyvznomq+e3btzPlMSz20jASx9Y0GyljJl7JmIlXNvQ0DMOohZnnTZS8vLwq0zqe\nqR40w4OYSW40JxqVeS4i60VkpYisEJHFrmyOiCx3r09FZHmtNr1FZKeI/CRQNlhEVonIRyJyT6C8\njetvnYi8IyK9A/eudfXXisiUQHkfEVnk7j3pDnFrMZSWloY21YNmuDl3DCM1wq5pVgL5qnqmqg4D\nUNXJqjpYVQcDTwPP1GpzJ7CgVtlsYKqq9gf6i8h4Vz4VKFXVfsA9wO0A7gTKm4GzgOFAgYh0dm1u\nA+50fW13fTQ49X0Oc3Merzm/Nxuv6RJWOxPqFrBXAVV7AUVkAvAJsDtQdjTQUVWXuKLHgMvwjuSd\nAPjJHOcC97nr8UChqpa7PgqBC4G/AqMBf3/ho8BM4IGQ7ydjFBcX1ziLuT7Gi7UV0yedMZb1+f4a\n4rO08ZruePVFWE1TgZdFZImIXBe8ISLnAptU9WP3e3vg50CEmknAewIbA79vdGX+vc8BVLUC73z0\nvGC5owToKSJdgTJVrQz01SPke2l2BE11M8MNI7OE1TRHquqXInIknvBco6pvunvXUPPs85nA3aq6\nRyTpNdkwDe1UBsMw6p+6NJRoLzwz+ifuOhvYBPQI3H8dzzT/BCgDtgI/AI4G1gTqTQZmu+uXgOGB\nPjcH6vwh0OYPwNXuejOQ5a5HAC/GmK/ay172ahmvROVZMq+4mqaItMMTTruc6T0Oz/QGGIsnCL/w\n66vqeYG2BcBOVb3f/V4uIsOAJcAUYJarOh+4FngXmAS85soXAr92zp8sN94Md6/I1f2ra/tctPnX\nRwiCYRgthzDmeXdgnoioq/+Eqha6e1dT0zSPxw3AI0AOsEBVX3LlDwGPi8g6YBuehomqlonIrcBS\nvG+SiKpud21mAHPc/RWuD8MwjIzS7IPbDcMw0kp9rAEk+8IzyZcD893vuUAhsBbPdO8cqPsLYB2w\nBhgXKB8MrAI+Au4JlLcB5rg27wC93Xgr3JgfAaXAF8A/8WJRO2VgvOWB8dbiLVv8J15sbF49jPeY\n6/M94LcZ/jzXu/srgMXA0DSO9zmw0vX9iau/zr2vTDwv0cbL5PMSbbxMPi+xxsvU8xJtvEw+L70D\n964Nfp5x5VJDC8Y4QnMa8L9UC83bgJ+76+n+Hw04xX2wrYA+wL+o1qLfBc5y1wuA8e76+8D97vpq\n94FOA/6GF1/aGS9+9GN3/VvgNxker4t7UF4BPsX9EwADMjTeJcCewDjdMjzeq3hCpTNwEVCUxs9z\nF/Pcbs0AAAPTSURBVN6Xam7gb3YP3nJPZ9L/vEQbL5PPS7TxMvm8RBsvk89LtPEy+bzMcde1P8+P\nCXy5Rns12ixHItIL+Drwp0DxBLxAdtzPy9z1pXgfwiFVXY/3bTKsjoD62n3NBS5w430IbFHVclV9\nDk+zvRBYBPTK8HjbgQq8aIIgEzIxHvBveP9wYwFUdWuGxzuAp6FciPeAlqTx82yLp9kGN0RcCDzv\nfqb7eTlsvAw/L9HGy+TzEu3zzOTzEm28TD4vo9117c/T//vFpDHv174b+BneN4BPd1X9CkBVN4nI\nUa68J57K7VPiyg4RMqBeRI4AbsX7Ftobpa98qp1eGRnP7aTahGdqBcnU++uPFxZ2m4h8D/ipqi7L\n4HjT8ITJMDzN4pw0vr9KvKiLXDzHIXhOzH8BPTPwvEQbL9hXPul9Xg4bT0QuJXPPS7T3l8nnJdp4\nmXxe6txAQx00Sk1TRC4GvlLVf1J3EHtavFhuvApgdYzxLgAOqmoikQIJjScibfHWaV5NxxjxxnO0\nwotkmIW3i+upDI/3feAFPGE6DXg4HeM5NuJt5f0TMMLtVKv9fKTT61l7vK8F7qX1eYkx3rnA/yUD\nz0sd42XkealjvEw+L0mHIjZKoQmMBC4VkU/wvq1Hi8jjwCYR6Q5Ve9k3u/olwLGB9r1cWazy2m2+\nBnTE+4a7BjhRRB5z98YAJwDfDPST9vHwTIk+wE144VS9gOVOOyrBc6yk+/19DmwBSpxJU+G2qGZi\nvMfxnBY73Xhz8RKx1G6b8Hgikg20V9VSvAX9Ejzt5CugH1CSzueljvEgA89LjPHOJ0PPSx3vLyPP\nSx3jZfJ56eTGizX32NS14NkYXngPh+8Iuh2YrrEdQW2AvtRcGPbVe8FbGL7Qlf+A6oXhyVQvDF9M\ntePiSrx1lb615pSp8fxF6S54C/u5GR5vGp7Htwue6fVZhsdbg/dAdsETLkvSNN4UYK677om3HHAZ\n1Y6gLqT3eYk1Xqael1jjZep5iTXeTWTmeYk1Xqael+DzGXQEVX2edcqkhhaKCQrNPLyF6LV4C7Zd\nAvV+4T682iEIQ/AWk9cB9wbKj8Dz7K5zH3SfwHgrXPkBvG2gfpjO/Rke7yNcyANe2EXtEJJMjPeW\na78UOD/D423A++degbcmdWaaxlsBfOB+vodnNq5z/a3OwPMSa7xMPS+xxsvU81LXeJl4XmKNl6nn\nper5dPf+T+3Ps66XBbcbhmEkQGNd0zQMw2iUmNA0DMNIABOahmEYCWBC0zAMIwFMaBqGYSSACU3D\nMIwEMKFpGIaRACY0DcMwEuD/AzWrfkZzCsUNAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x6487080>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "v = vs[0]\n",
    "v.network().as_dataframe().plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Describing the PEST 'Job'\n",
    "\n",
    "When configuring PEST to work with a particular model, you 'teach' PEST how to communicate with the model by describing the format of one or more text input files expected by the model and one or more text output files produced by the model. PEST will then generate updated input files for each simulation and read the updated output files that result from the simulation.\n",
    "\n",
    "veneer-py has some basic functionality for setting up PEST runs that avoids the need to directly edit the PEST configuration files.\n",
    "\n",
    "With veneer-py, you can describe a PEST job in Python, including describing the Source model parameters that you want to calibrate and the outputs that you want to calibrate against.\n",
    "\n",
    "veneer-py will then write out the following PEST configuration files and invoke PEST:\n",
    "\n",
    "* **PTF:** A PEST Template File, which describes how to run the model, including setting relevant model parameters and ensuring that the required model outputs are produced. PEST will substitute model parameters into this file, based on the configuration in the PCF. In the case of veneer-py, the PTF is a template of a Python script, which uses veneer-py to connect to a Source/Veneer server, sets parameters, runs the model and then compares results using defined statistics.\n",
    "* **PIF:** A PEST Instruction File, which tells PEST where to find the observations that it needs for evaluating a simulation. In the case of veneer-py, these outputs are produced by the logic in the PTF.\n",
    "* **PCF:** A PEST Control File, which desribes the parameters to be optimised and the observations to optimise to, as well as how to run the model.\n",
    "* **PRF:** (OPTIONAL), used when performing a parallel calibration with more than one Veneer/Source server. Describes where (on the filesystem) to run the model from.\n",
    "\n",
    "To establish all these files and run a PEST job, you can use functionality in veneer-py to describe a PEST 'Case'.\n",
    "\n",
    "The 'Case' will ultimately know everything about the calibration\n",
    "\n",
    "* what parameters you are calibrating and how they are constrained,\n",
    "* what you are calibrating to,\n",
    "* how many Source servers you have at your disposal\n",
    "* any options related to PEST, such as which optimisation routine to use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from veneer.pest import Case"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At very least, we need to give a Case a `name` - which is the basis for all the filenames that will be written out.\n",
    "\n",
    "You can also specify:\n",
    "\n",
    "* an optimiser (the default is `pest`, but the PEST software also comes several others)\n",
    "* a list of Veneer/Source servers, described as a list of ports\n",
    "* a random number seed in order to either make the optimisation deterministic or allow random variation\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "calibration = Case('CalibrationCase',optimiser='cmaes_p',model_servers=ports)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PEST has many options - most of which we leave at default. One option that we currently need is to put PEST into single precision mode. This is because PEST, in double precision mode, uses a syntax for floating point literals that is not valid Python:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "calibration.options['PRECIS']='single' "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configuring the calibration parameters\n",
    "\n",
    "PEST needs to be told about the calibration parameters\n",
    "\n",
    "This is a two step process:\n",
    "* Specify how to apply the parameter (using a statement as you would in Veneer. eg `model.catchment.runoff.set_param_values('baseflowCoefficient',0.5,fus=list(fu_types))` but with the actual value (`0.5`) changed to the PEST parameter name, with markers (eg `@v_bfCoeff@`). This information forms part of a dynamically generated Python script that PEST will modify and run for each simulation\n",
    "* Tell PEST about the parameter, including its range.\n",
    "  * This forms part of the PEST control file\n",
    "  * In addition to the range, we also specify the initial value.\n",
    "  * We set the initial value to be halfway between the min and max\n",
    "\n",
    "We're performing a rainfall runoff calibration, with the four parameter GR4J rainfall runoff used in the model\n",
    "\n",
    "We're going to perform a lumped calibration - ie one parameter set everywhere - but we could, alternatively, calibrate distinct parameters by functional unit type, or in different parts of the catchment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['TIME.Models.RainfallRunoff.GR4J.GR4J']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "v.model.find_model_type('GR4J')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['C', 'k', 'x1', 'x2', 'x3', 'x4']"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = v.model.find_parameters('TIME.Models.RainfallRunoff.GR4J.GR4J')\n",
    "params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We only want to calibrate `x1`-`x4` - (`C` and `k` are specific to the eWater version of GR4J - they provide a baseflow filter)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['x1', 'x2', 'x3', 'x4']"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "params = params[2:]\n",
    "params"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to assign ranges to each of these. The model implementation in Source has metadata about suitable ranges - but at this stage, there isn't an easy way to interrogate that information from veneer-py. You can check in the Source user interface (`Edit|Rainfall Runoff Models`) to see the ranges.\n",
    "\n",
    "Having done that, we'll construct a Python dictionary of parameter ranges\n",
    "\n",
    "**Note:** These are quite narrow ranges for GR4J. This is done to keep the optimisation short in the context of the tutorial (and because the 'observed' data is actually synthetic data generated from Source)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'x1': [100.0, 500.0], 'x2': [1.0, 5.0], 'x3': [1.0, 200.0], 'x4': [0.5, 3.0]}"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ranges = {\n",
    "    'x1':[100.0,500.0],\n",
    "    'x2':[1.0,5.0],\n",
    "    'x3':[1.0,200.0],\n",
    "    'x4':[0.5,3.0]\n",
    "}\n",
    "ranges"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we can loop over each parameter and 'teach' PEST about it - ie tell PEST how to modify the parameter and tell PEST what range we want to calibrate over:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Configuring x1\n",
      "Configuring x2\n",
      "Configuring x4\n",
      "Configuring x3\n"
     ]
    }
   ],
   "source": [
    "for param,param_range in ranges.items():\n",
    "    print('Configuring %s'%param)\n",
    "\n",
    "    pest_pname = '$'+param+'$'\n",
    "\n",
    "    # 1. Tell PEST how to set the parameter\n",
    "    calibration.parameters.model.catchment.runoff.set_param_values(param,pest_pname)\n",
    "\n",
    "    # 2. Details of the PEST parameter. name, starting value, min, max\n",
    "\n",
    "    # Decide what to use for the initial value... half way between min and max!\n",
    "    initial = 0.5*(param_range[0]+param_range[1])\n",
    "    \n",
    "    calibration.parameters.describe(pest_pname,initial,param_range[0],param_range[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** When we tell PEST how to set the parameter in the model, we use a Python statement that looks similar to statements from earlier sessions:\n",
    "```python\n",
    "calibration.parameters.model.catchment.runoff.set_param_values(param,pest_pname)\n",
    "# is similar to\n",
    "v.model.catchment.runoff.set_param_values(param,value)\n",
    "```\n",
    "The instructions to PEST will in fact translate to instructions to a Veneer client (`v`). So everything after `.parameter.` should be something you can call on a Veneer client. The other main difference is that, instead of passing in an actual value at this point, you pass `pest_pname`, which will be something like `$x1$` and will get translated to an actual value at runtime.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The previous code has gone some way towards configuring the PEST job. We can get a preview of what has been achieved by asking to see the configuration as it stands.\n",
    "\n",
    "First, the state of the PTF (PEST Template File). You can see that it looks very much like a Python script, except where it has references to things like `$x1$` and the like"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ptf $\n",
      "from veneer.pest_runtime import *\n",
      "from veneer import Veneer\n",
      "from veneer.stats import * \n",
      "import pandas as pd\n",
      "from veneer import general\n",
      "\n",
      "general.PRINT_URLS=False\n",
      "veneer_port=find_port()\n",
      "v = Veneer(port=veneer_port)\n",
      "v.model.catchment.runoff.set_param_values('x1',$x1$)\n",
      "v.model.catchment.runoff.set_param_values('x2',$x2$)\n",
      "v.model.catchment.runoff.set_param_values('x4',$x4$)\n",
      "v.model.catchment.runoff.set_param_values('x3',$x3$)\n",
      "# Run Model\n",
      "v.drop_all_runs()\n",
      "v.run_model(**{})\n",
      "\n",
      "observed_ts={}\n",
      "pest_observations=[]\n",
      "\n",
      "# Get results\n",
      "run_results = v.retrieve_run()\n",
      "\n",
      "# Compute stats\n",
      "\n",
      "\n",
      "\n",
      "# Write summary results\n",
      "print(pest_observations)\n",
      "write_outputs(pest_observations,'_CalibrationCase_output.txt')\n",
      "write_outputs(pest_observations,'__outputs_to_keep.txt')\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(calibration.ptf_text())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are still gaps in the PTF - eg the `# Compute Stats` section - that will come as we describe the outputs and observations.\n",
    "\n",
    "The PCF (PEST Control File) is also partly complete:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pcf\n",
      "* control data\n",
      "restart estimation\n",
      "4 0 1 0 1  \n",
      "1 1 single point    \n",
      "10.0 2.0 0.3 0.03 8   \n",
      "10.0 10.0 0.001   \n",
      "0.1     \n",
      "50 0.005 4 4 0.005 4   \n",
      "1 1 1       \n",
      "* singular value decomposition\n",
      "1\n",
      "4 5e-07\n",
      "0\n",
      "* parameter groups\n",
      "default_pg relative 0.001 0.0001 switch 1.5 parabolic   \n",
      "* parameter data\n",
      "x1 none factor 300.0 100.0 500.0 default_pg 1.0 0.0 1\n",
      "x2 none factor 3.0 1.0 5.0 default_pg 1.0 0.0 1\n",
      "x4 none factor 1.75 0.5 3.0 default_pg 1.0 0.0 1\n",
      "x3 none factor 100.5 1.0 200.0 default_pg 1.0 0.0 1 \n",
      "* observation groups\n",
      "default_og  \n",
      "* observation data\n",
      "\n",
      "* model command line\n",
      "python _run_CalibrationCase.py\n",
      "* model input/output\n",
      "_run_CalibrationCase.tpl _run_CalibrationCase.py\n",
      "_CalibrationCase_output.ins _CalibrationCase_output.txt\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(calibration.pcf_text())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** There are a lot of options in the PCF - and we are using defaults. They are very well described in the PEST Manual and can be specified using `calibration.options`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "OrderedDict([('NPAR', None), ('NOBS', None), ('NPARGP', 1), ('NPRIOR', 0), ('NOBSGP', 1), ('MAXCOMPDIM', ''), ('DERZEROLIM', ''), ('NTPLFLE', None), ('NINSFLE', None), ('PRECIS', 'single'), ('DPOINT', 'point'), ('NUMCOM', ''), ('JACFILE', ''), ('MESSFILE', ''), ('OBSREREF', ''), ('RLAMBDA1', 10.0), ('RLAMFAC', 2.0), ('PHIRATSUF', 0.3), ('PHIREDLAM', 0.03), ('NUMLAM', 8), ('JACUPDATE', ''), ('LAMFORGIVE', ''), ('DERFORGIVE', ''), ('RELPARMAX', 10.0), ('FACPARMAX', 10.0), ('FACORIG', 0.001), ('IBOUNDSTICK', ''), ('UPVECBEND', ''), ('ABSPARMAX', ''), ('PHIREDSWH', 0.1), ('NOPTSWITCH', ''), ('SPLITSWH', ''), ('DOAUI', ''), ('DOSENREUSE', ''), ('BOUNDSCALE', ''), ('NOPTMAX', 50), ('PHIREDSTP', 0.005), ('NPHISTP', 4), ('NPHINORED', 4), ('RELPARSTP', 0.005), ('NRELPAR', 4), ('PHISTOPTHRESH', ''), ('LASTRUN', ''), ('PHIABANDON', ''), ('ICOV', 1), ('ICOR', 1), ('IEIG', 1), ('IRES', ''), ('JCOSAVE', ''), ('VERBOSEREC', ''), ('JCOSAVEITN', ''), ('REISAVEITN', ''), ('PARSAVEITN', ''), ('PARSAVERUN', ''), ('SVDMODE', 1), ('MAXSING', None), ('EIGTHRESH', 5e-07), ('EIGWRITE', 0), ('SIM_OPTIONS', {})])"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "calibration.options"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configuring the outputs and observations\n",
    "\n",
    "PEST needs to know what its calibrating to. In our case, that means a time series of observed data, and a corresponding modelled time series. We also need the objective function and the time period for the calibration.\n",
    "\n",
    "PEST has a tool for processing time series data (TSPROC) but we don't use it here. Rather, we compute the objective values in Python and just pass single numbers back to PEST.\n",
    "\n",
    "We'll start by loading the observed flow time series data into this notebook and exploring the data a bit...\n",
    "\n",
    "**Note**: Loading the data at this point serves two purposes:\n",
    "1. We become more familiar with the data and can identify things like the time period we want to calibrate over, and\n",
    "2. We work out the exact `pandas` command needed to load the data. We need to give this command to PEST later on to call as part of the simulations..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Flow</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Date</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2007-01-01</th>\n",
       "      <td>0.000000e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2007-02-20</th>\n",
       "      <td>1.103490e+00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2007-04-11</th>\n",
       "      <td>1.305575e+03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2007-05-31</th>\n",
       "      <td>3.417483e+03</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2007-07-20</th>\n",
       "      <td>1.441880e+04</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2007-09-08</th>\n",
       "      <td>1.650685e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2007-10-28</th>\n",
       "      <td>7.711668e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2007-12-17</th>\n",
       "      <td>3.111951e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008-02-05</th>\n",
       "      <td>2.460641e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008-03-26</th>\n",
       "      <td>3.217175e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008-05-15</th>\n",
       "      <td>2.234289e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008-07-04</th>\n",
       "      <td>2.927276e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008-08-23</th>\n",
       "      <td>1.932046e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008-10-12</th>\n",
       "      <td>1.772153e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2008-12-01</th>\n",
       "      <td>3.613313e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2009-01-20</th>\n",
       "      <td>2.899430e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2009-03-11</th>\n",
       "      <td>1.726069e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2009-04-30</th>\n",
       "      <td>1.006491e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2009-06-19</th>\n",
       "      <td>1.679338e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2009-08-08</th>\n",
       "      <td>1.441411e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2009-09-27</th>\n",
       "      <td>5.468952e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2009-11-16</th>\n",
       "      <td>7.615090e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-01-05</th>\n",
       "      <td>4.580939e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-02-24</th>\n",
       "      <td>2.181879e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-04-15</th>\n",
       "      <td>1.539815e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-06-04</th>\n",
       "      <td>1.660170e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-07-24</th>\n",
       "      <td>1.330355e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-09-12</th>\n",
       "      <td>3.419601e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-11-01</th>\n",
       "      <td>3.786437e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2010-12-21</th>\n",
       "      <td>3.170745e+06</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    Flow\n",
       "Date                    \n",
       "2007-01-01  0.000000e+00\n",
       "2007-02-20  1.103490e+00\n",
       "2007-04-11  1.305575e+03\n",
       "2007-05-31  3.417483e+03\n",
       "2007-07-20  1.441880e+04\n",
       "2007-09-08  1.650685e+05\n",
       "2007-10-28  7.711668e+06\n",
       "2007-12-17  3.111951e+06\n",
       "2008-02-05  2.460641e+06\n",
       "2008-03-26  3.217175e+06\n",
       "2008-05-15  2.234289e+06\n",
       "2008-07-04  2.927276e+06\n",
       "2008-08-23  1.932046e+06\n",
       "2008-10-12  1.772153e+06\n",
       "2008-12-01  3.613313e+06\n",
       "2009-01-20  2.899430e+06\n",
       "2009-03-11  1.726069e+06\n",
       "2009-04-30  1.006491e+06\n",
       "2009-06-19  1.679338e+06\n",
       "2009-08-08  1.441411e+06\n",
       "2009-09-27  5.468952e+06\n",
       "2009-11-16  7.615090e+06\n",
       "2010-01-05  4.580939e+06\n",
       "2010-02-24  2.181879e+06\n",
       "2010-04-15  1.539815e+06\n",
       "2010-06-04  1.660170e+06\n",
       "2010-07-24  1.330355e+06\n",
       "2010-09-12  3.419601e+06\n",
       "2010-11-01  3.786437e+06\n",
       "2010-12-21  3.170745e+06"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "flows = pd.read_csv('SyntheticObservedFlow.csv',parse_dates=True,dayfirst=True,index_col=0)\n",
    "flows[0::50] # Show every fifty days"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0xf489080>"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWkAAAEVCAYAAAAimxT0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3Xl8VPW5P/DPEwirgCwCsoOiIooLSi0uDKWKyq1y664F\n9SpqrcWl9me192pqq9af3lastha3Sm0V16pVK24TlYqCgCiLosgiO0ZIwhKSzHP/eOZwJpNJMjP5\nnsyc5PN+vfLKZObMme+cnPnMc77ne84RVQUREeWnglw3gIiI6saQJiLKYwxpIqI8xpAmIspjDGki\nojzGkCYiymOBhLSIPCwiG0VkURrT/k5EFojIfBH5TERKgmgTEVEYSRDjpEXkOADlAGao6ogMnncV\ngMNV9VLnjSIiCqFAKmlVfQ/At4n3icgQEXlVROaKSLGIHJDiqecBeCKINhERhVHrJnyt6QAuV9Uv\nRWQUgD8BGOc9KCIDAAwC8FYTtomIKK81SUiLSEcAowE8LSISv7swabJzATyjPE6diGiPpqqkCwB8\nq6pH1jPNuQCubKL2EBGFQkZ90iLSRUSeFpGlIrJYRL5T3+TxH6hqGYCvROTMhHmNSLh9EIC9VXVO\nZs0nImreMt1xOA3AK6o6DMBhAJammkhE/g7g3wAOEJHVInIxgAsAXCIiC0XkUwCnJTzlHABPZtx6\nIqJmLu0heCLSGcACVd0v2CYREZEnk0p6MIAtIvJo/MCT6SLSPqiGERFRZiHdGsCRAO6P7wDcAeAX\ngbSKiIgAZDa642sAa1R1XvzvZwDckDyRiHAIHRFRFlRVku9Lu5JW1Y0A1iQcKTgOwJI6pnX6c8st\nt+T1/IKcbxjmGYY2tvT3znnm/zzrkuk46akA/iYihQBWALg4w+dnJRKJ5PX8gp6va2FYni11WYZp\nnkEIy3tvyuXp/ARLIqKu59mSFRUVoaioKNfNaDa4PN3hsnRLRKCN6e6ghnXuDHz0kdt5hqUCCgsu\nT3e4LJsGK2mHRIDp04EpU3LdEiIKm7oq6aY8C16LILUWMVHLM2jQIKxatSrXzchLAwcOxMqVK9Oe\nniFNRM6tWrWq3hELLZlkWMmxT9oxVtJE5BJD2jGGNBG5xJAmIspjDGnHWEkThdOqVatQUFCAWCyW\n66bUwJB2jCFNlP8GDRqEDh06oHPnzujUqRM6d+6MdevWZbxTrykwpImoxRERvPzyyygtLUVZWRlK\nS0vRp0+fXDcrJYa0Y3n4RUxEKTQ0RHD9+vU4/fTT0b17dxxwwAF46KGHAAAVFRXo0KEDSkpKAAC3\n3XYbCgsLUV5eDgC4+eabcd111zlrJ0PaMYY0UfNwzjnnYMCAAdiwYQOefvpp3HTTTYhGo2jbti1G\njRqF4uJiAMA777yDQYMGYfbs2QCA4uJijBkzxlk7GNJElBMibn6yNXHiRHTr1g3dunXDD3/4wxqP\nrVmzBu+//z7uvPNOFBYW4rDDDsOll16KGTNmAABOOOEEFBcXo7q6GosWLcLUqVNRXFyMiooKzJ07\nFyeccEJjFk0NDGnHWEkTpUfVzU+2XnjhBZSUlKCkpATPPfdcjcfWr1+Pbt26oUOHDnvuGzhwINau\nXQsAGDNmDN5++23Mnz8fI0aMwIknnohoNIo5c+Zg6NCh6Nq1a/YNS8KQJqIWqb4+6T59+qCkpATb\nt2/fc9/q1avRt29fAMDo0aPx2Wef4fnnn8eYMWNw0EEHYfXq1XjllVecdnUADGkioj284O7Xrx9G\njx6NG2+8ERUVFVi0aBEefvhhTJo0CQDQvn17jBw5Evfff/+eUB49ejQeeOABhnS+Y3cHUf6razx0\n4v1PPPEEvvrqK/Tp0wdnnHEGfv3rX2Ps2LF7Hh8zZgyqq6sxatSoPX+Xl5c77Y8GeD5pp0SAxx8H\nLrgg1y0hyq34uZFz3Yy8VNey4ZVZiIhCiCHtGLs7iMglhrRjDGkicokhTUSUxxjSjrGSJiKXGNKO\nMaSJyCVeiJaInBs4cGBenps5HwwcODCj6RnSjnG9JAJWrlyZ6yY0G+zucIwhTUQuMaSJiPJYRt0d\nIrISwDYAMQCVqjoqiEaFGStpInIp0z7pGICIqn4bRGOaA4Y0EbmUaXeHZPEcIiLKUqaBqwBeF5G5\nIjIliAaFFU/4RURByLS741hVXS8i+8DCeqmqvhdEw8ImFrPfDGsicimjkFbV9fHfm0XkeQCjANQK\n6aKioj23I5EIIpFIoxoZBl44M6SJKB3RaBTRaLTB6dI+6b+IdABQoKrlItIRwCwAv1LVWUnTtciT\n/u/eDbRtC8ycCZx9dq5bQ0RhU9dJ/zOppHsBeF5ENP68vyUHdEvmdXd4v4mIXEg7pFX1KwCHB9iW\nUGN3BxEFgcPpHGNIE5FLDGlHWEkTURAY0o4xpInIJYa0I6ykiSgIDGnHGNJE5BJD2hFW0kQUBIa0\nYwxpInKJIe0YQ5qIXGJIO8LuDiIKAkPaMYY0EbnEkHaElTQRBYEh7RhDmohcYkg7wkqaiILAkHaM\nIU1ELjGkHWNIE5FLDGlH2N1BREFgSDvGkCYilxjSjrCSJqIgMKQdY0gTkUsMaUe8cOaFaInIJYa0\nY6ykicglhrRjDGkicokh7Qh3HBJREBjSjjGkicglhrQjrKSJKAgMaccY0kTkEkPaEVbSRBQEhrRj\nDGkicokh7QgraSIKQsYhLSIFIjJfRF4MokFhx5AmIpeyqaSvBrDEdUOaC4Y0EbmUUUiLSD8ApwJ4\nKJjmhBe7O4goCJlW0r8H8HMAjKI6MKSJyKXW6U4oIhMAbFTVhSISASB1TVtUVLTndiQSQSQSyb6F\nIcFKmogyEY1GEY1GG5xONM1UEZHbAfwIQBWA9gA6AXhOVScnTafpzrM52bAB2Hdf4NZbgf/5n1y3\nhojCRkSgqrWK37S7O1T1JlUdoKpDAJwL4K3kgG7JWEkTURA4TtoxhjQRuZR2n3QiVS0GUOy4Lc0C\nQ5qIXGIl7Qi7O4goCAxpxxjSROQSQ9oRXoiWiILAkHaMlTQRucSQdoR90kQUBIa0YwxpInKJIe0I\nK2kiCgJD2jGGNBG5xJB2jCFNRC4xpB1hdwcRBYEh7RhDmohcYkg7wkqaiILAkHaMIU1ELjGkHWEl\nTURBYEg7xpAmIpcY0o4xpInIJYa0I+zuIKIgMKQdY0gTkUsMaUdYSRNREBjSjjGkicglhrQjrKSJ\nKAgMaccY0kTkEkPaEVbSRBQEhrRjvBAtEbnEkHaMlTQRucSQdoTdHUQUBIa0YwxpInKJIe0IK2ki\nCkLrdCcUkbYA3gHQJv68Z1T1V0E1LKwY0kTkUtohraoVIjJWVXeISCsAs0XkVVX9MMD2hQYraSIK\nQkbdHaq6I36zLSzgGUlJGNJE5FJGIS0iBSKyAMAGAK+r6txgmhVeDGkicint7g4AUNUYgCNEpDOA\nf4jIwaq6JHm6oqKiPbcjkQgikUgjm5n/2N1BRJmIRqOIRqMNTieaZaqIyP8A2K6qv0u6X7OdZ5gt\nWwYMGwZccAHw+OO5bg0RhY2IQFUl+f60uztEpIeIdInfbg/gRADL3DUx3FhJE1EQMunu2BfAYyJS\nAAv3mar6SjDNCi+GNBG5lMkQvE8AHBlgW0KNlTQRBYFHHDrGkCYilxjSjvFUpUTkEkPaEa+Crq7O\nbTuIqHlhSDtWVZXrFhBRc8KQdsSrpBnSROQSQ9oxhjQRucSQdoSVNBEFgSHtGEOaiFxiSDvC0R1E\nFASGtGOspInIJYa0YwxpInKJIe0IdxwSURAY0g6JMKSJyC2GtCOqQOvWDGkicosh7VBhIUd3EJFb\nDGlHWEkTURAY0g4VFjKkicgthrRDrKSJyDWGtCPs7iCiIDCkHWJIU1iUl9uQUcp/DGlHvEqaozso\nDHbsyHULKF0MaYe445DCwquieeHk/MeQdoR90hQmPGtjeDCkHWIlTWHhhTPX1/zHkHZEFSgosN+x\nWK5bQ1Q/L5wrK3PbDmoYQ9ohEXZ5UDh4lTRDOv8xpB3jCA8KA4Z0eDCkHfF2xLCSpjBgd0d4pB3S\nItJPRN4SkcUi8omITA2yYWHE7g4KC+44DI/WGUxbBeA6VV0oInsB+EhEZqnqsoDaFiqspClM2N0R\nHmlX0qq6QVUXxm+XA1gKoG9QDQsjEaBVK4Y05T92d4RHVn3SIjIIwOEAPnDZmDBLrKR3785tW4ga\nwko6PDLp7gAAxLs6ngFwdbyirqWoqGjP7UgkgkgkkmXzwkUE6NMHWLcOGDgw160hqhtDOvei0Sii\n0WiD04lmcPC+iLQG8E8Ar6rqtDqm0Uzm2VzMmQNccw2w337A+PHA5Mm5bhFR3WbPBo47Dnj/feCY\nY3LdGgIAEYGq1jo3YabdHY8AWFJXQLdk3vfS/vsDy5fnti1EDWElHR6ZDME7FsAFAL4nIgtEZL6I\nnBxc08JHBOjdG9iyJdctIaofQzo80u6TVtXZAFoF2JZQ8yrpdu2AXbty2xaihnB0R3jwiEOHRBjS\nFA7eScAY0vmPIe0IK2kKE4Z0eDCkHWIlTWHhhTQPvMp/DGlHWElTmHjrKyvp/MeQdowhTWHA7o7w\nYEg7xO4OCovkkBYBFi7MXXuobgxpR9jdQWGSqpKePz83baH6MaQdYiVNYeEVFYk7DktLc9MWqh9D\n2hFW0hQmqSpphnR+Ykg7xEqawiJVSO/cmZu2UP0Y0o6wkqYwSTUEjyM98hND2iERoG1boKLC/xAQ\n5aNUlTQvVpGfGNKOFRQAhYVc4Sm/pTrikOtsfmJIO5JYObPLg/Jdqkqa3R35iSHtkMSvqcCQpnyX\nqk+alXR+Ykg7wkqawoR90uHBkHbIq6Tbt2/akL7vPmDt2qZ7PQo/hnR4MKQdyWUl/dOfAtN41UnK\nALs7woMh7VAu+6S/+aZpX4/CLRYD2rSx0R1eVV1Rkds2UWoM6QC0awdcey3w6KNN95qtePVJykAs\nBnToYEcZehel5RGH+Ykh7Ugs5lfSHTsC778PXH11071+YWHTvRaFnyrQuTNQXu6PlWZI5yeGtCPV\n1X4127mz/S4rC/51eWQjZSMWA7p0sZBmJZ3fGNKOxGK1Q7opeB8w7vShTMRitp6WlbGSzncMaUeq\nq+2QcKBpQ5ofMMqGF9KspPMfQ9qRVN0dTYEhTdnw+qTLymzdbdsW2LEj162iVBjSjuSqu8Mb58qQ\npkwk9klXVQGdOnEdylcMaUdy3d3Bw9ApE7GYjULatcv2Z7RrZ/fzJEv5hyHtCLs7KExUbX3t2BHY\nts1ue+OmKb9kFNIi8rCIbBSRRUE1KKwSuzu6dWu612V3B2UjFrMtv06dgG+/9QO7vDzXLaNkmVbS\njwIYH0RDwi6xu2Pfff37E0+qHgR2d1A2vJDu3h3YtAlo3dpu8/QC+SejkFbV9wB8G1BbQi2xku7f\n3zYdO3UCtm8P9nUrK+0cDKykKRPeEbI9egAbN9q626MHQzofsU/akcRKumNHC+dOnYDS0mBft6rK\n+sAZ0pQJVb+S3rjRr6S3bMl1yygZQ9qRxB2Hnl697AMQpKoq+3Bt3cpDxCl9XndHjx7Ahg227rK7\nIz+1DmKmRUVFe25HIhFEIpEgXiavJHZ3eHr3tg9AkCorrXLv0MGCumvXYF+PmofE7o6PP/a7O1hJ\nN51oNIpoNNrgdNmEtMR/6pQY0i1FYneHJzGkRYB584CRI92+blWVbarusw+weTNDmtLjFRVdu9bs\n7kh1hZ/584Ejj2z6NjZ3yQXsr371q5TTZToE7+8A/g3gABFZLSIXN6KNzUqq7g4vpL2Tqs+e7f51\nq6rsNKU9e9peeqJ0VFdbMCd2d6SqpFWtsGCFnTsZVdKqen5QDQm7VN0dffvapqR3ytLVq92/bmWl\nfdj23tsqaaJ0eFtg3uiOQYNS7zj0dkjzqi25wx2HjqTq7jjgAODzz+1gAaDmTpmlS928rldJ77MP\nK2lKn/flvs8+FsCtWqXeGtu61X4HPZSU6saQdiRVd8dBBwGffeav4M8+a9V0aSlw8MF+N0hjeBVR\nz56spCl93pd7//72d6tWdnvNmprTbdtmvxnStlXh4jObKYZ0I3z9tX+7ru6OsjJ/GF5ZGXDzzf4m\npYsrtyRWREEP96PmI3GHM2DVdK9eVjknjrn3bjOkbQTVbbc1/esypBuhf38/qFN1dxQUAEOHWr/0\nkCF2X0WFX/F6VUpjeBVRnz7AunWNnx+1DF5Ie9fl3LTJ1td+/WoWH965YXhODzN/ftO/JkO6kbxK\nI1V3B2A79O6/33bMjB1r03tHIXr9fY3hfdgGDwa++qrx86OWwVtvPF6lPHBgzR3cXkizkja5WA4M\naUfqCunTTgO+/BJo3x74zW9suJO3p9xFSJ9zjs2HIU2pVFamvv6l103m8cJnwICaIe2dwIshbTJd\nDitW2Fj0xmw1M6SzlHwIdmWldTsku+Ya+71rl42bXr/eP2Odi5AGrI+7e3f7QLmaJzUPxx8PnHpq\n7fsTK+lHHgEeeMBuDxkCLF/uT8dKuqZMl8Pnn9tncu+9s39NhnSWvArDW4l377az0SXz+vy+/tr6\nsLds8fegN7ZP2mtDYaG9DqtpSvbBBzVD1+PtywCAiy8Gzj7bbh96KPDJJ/509YX0qlX+RWxbilTL\nYdOmunfae1e8AbJfVgzpLHkr79tv+3+nCmlP1672ofjhD4Fp0+y+xp7MxtuZ47Vlv/2AL77wH1+7\nFnjnnca9BoVfqnBI7pP2jBgBLFpUczog9UikQYOABx900sTQSBXSI0cCRxyRevrE88lnO0SWIZ0l\nLxivusr6mHfvTt3dAVgl889/2u0zz7QKRAT49NPGtcHbAem15dBDa37ATj8dGDOmca9B4ZfqwhN1\nhfTgwbbfxNsi89atp59OfdRhSYm7doaBF9KVlfYZjsVsK7muE6klXoE91XlR0sGQzlLiBTs3b66/\nkt5/f+szBoBDDrHfo0c3fjiPtwJ4O4YOO8yG+3nqO8f0vHnAK6807vUpHFJdXDZ5x6GnoAC45BJ/\na6+qyk6utHQp8PvfB9vOMPBC2tu56h1NrJr6SywxpL0hspdcAqxcmf5rMqSzlBzS9VXSiQYPtt/D\nhgHLljXunAheOF9/vf1ODukOHep+7k03ARMmZP/a+US15hYE1ZSqkq5rRzcATJoEvPuuP92wYcDE\niS17f4eqVc7t21tF7O1XWr8eOPBAu/3ee7WflxjS3vjzRx4BZs1K/7UZ0llKDOlNm+recZisoMCu\npHLCCbbyf/hh9m3Yvdv6w6ZMsb+HDLHNT+9se/WFdNu22b9uvlm0yL6gKLVUQ/DKyuzKQakccYQV\nEDt2+GH+k5/YKQ483uHRqebdHHlbHueea0cdJoZ0hw72OUxcPp4dO6wb8uabgQUL/P0DqbZi6sKQ\nzlLiyul1d6RTSQM2qmPSJODEE4HXX089zUsv2WZRfSoqan4xFBQAe+0FHHecffN7K0Kqaj3dtoaB\nN+yQV6apraAg9UWKy8qsWEilXTtg+HDrEvP6rocPt1EfXsh465SLo2bDYOdOWy5FRcCf/gRMnmz3\nb9hgXSBHHmnD7QDgoYeAceOAJUts+YwfbwMGiov9nf3fZnClWIZ0lhIr6S++SL+STlRfSBcV2WZR\nfXbvrl0RL1liH741a/xNrVQ7NbwAD/pq5k3B23nFQ5drq2t8bmlp3SENAN//PvCvf/nr9b772qkH\nvC0/12P9893OndbV0bcvcOut/v2TJ1s4H3ec393xj38Ab71lX2z33WfPO/RQG37rVduZnLGSIZ2l\nxJB+7DGrLDKtTo89Fli8OPXQnHQ2h5IracCG+h17rO2ULCuz06W++Wbt53o7FZPPepapOXNS75hq\nSt6XUUsJjEzstZf99vpDP/7Y+pZLS+vu7gDsSNkXXqjZLTJhgm3hAX4l3RKuiThnjlXB3pjnG2+0\n30cd5U8zfrx9lj79tGY3Y0mJ/V1QYMvv8cft/kwuq8eQzlJiMPXubeOlM710Vbt29mF48snaj6VT\nlaeqpAHgpJOA55+3D9All9g3e7KyMnv9yZMzG2S/dWvN6b/7XWDGjPSfH4SWtumdiaoqCxBvHTv8\ncNv03rHDD/BURo2yLZN33/VD+vzz7X9dVeWfg/q995p/v/SZZ1pftLe8WrcGrrvORsA8/7zd16MH\ncMMNduqHxM/k7t1+aJ91lm0dt25teZHuaU8Z0lnyQnrvvYHf/tb+GQMGZD6fyZOBv/yl9v0dO9rv\n+vpZ6wrp886zKmjDBuCii+yDtn69PVZSAsycaZXB7bfbhyyNa2Hu0bUrcNddNe/LpH8tCI09F8qm\nTc33ggm7dwNXXGHh4Y3L37LF1q/kszYm8obivfKKH9IjRtgJmF56yZb54MG2pTZuXPDvIxcqKuzH\n+4wlFmH/+782jPY//gP46CP7wrroIusiSt469cL9pJOs/3rffS03iovTawdDOkve0KTPPrMDRqJR\n66/K1LhxtumZfGSgd7Km+sIjVXcHYOcF9oYF9expJ2GaPt1ep3t34Npr7bFTTgFuucXfhE3XihU1\n/04cZpQLiSGdvJMsFrMgSdX3vmKFbbqecoots1yc0D1oFRV29sXrrrNgAazro77+aM9//Zf9Tpz2\nqquAe++1+bZrB7zxhoX/mjW2Lt1zT815iNiohjA691wgEvG7MRMP8fa0bu1fpLdbN1uXiovt2Agv\n1IcOtd9t29qW88iR9sXpjTu/4ooGvuhU1emPzbL5e/NN1UjEzbweflj1e99TnTdPdds2uy8SUd1n\nH9U771QdNUp1yZLaz/vjH1Uvvzz1PJcsUb3vPru9eLHN66WXVK02t5+1a1VXr1bt1k113br02gqo\nTplit3ftsr+vuSaz9+vaHXdYO6ZOtd/bt/uPrVpl961caX9XValu3Wq3//AHe+ykk+z3yy83fduD\nVlho/6eNG1V79vT/9x07pvf8t95S3bHD/7uiQnXwYNW771YdOdLu+8lPVH/2M5tv587+tJWVdt8T\nT7h7P02pUydrf48eqvvuqzppUsPP+eQTe86GDbasvvpKNRbzH4/FVKurbZn27q26YIH/OvHsrJWp\nrKSzlO7BK+mYNMkqkaOOAqZOtfvKyqzyueEG26P+6qu1n/fNN/6RjMmGDbOxrYBdqusHPwB+/GP/\n8TvvtM2u/v1ts9Y7ICYdXuXqdXPMm5fb4W9etXjvvfb33Ln+uSZWrbLf3oEYt97qj3jwdp5+/bWN\nZnjttaZrc1OorrYtiDZtbIvqj3+09WLqVOCYY9Kbx9ixNjrB06aNdZNdf73fJfff/w08+qjdLi31\nT4Hg7VRM3En2/vuNP5S8stLtzupU6+6gQbYOtW1r69eKFf57rM8hh1g29Oply2rQIP8ka4DdLiiw\nZXrLLbZlUt8OXIDdHVlr6IRKmSgstL2+p5xim4+lpbYijx5tO2oGDwaeeKL2iumdojQdt99uYXTw\nwfb3j37krzxFRdav9re/1XzOo4/W7ALwdhh6O+i2brUDaLZutQDIlYoK21y85RYLn0jENtFjMf+L\nxDsMd/Fi/3leiCxZAvziF8Df/968jlz0ho15/+czzrD3Om1a3UM/03HOOTVHH/XubcUEAPz618DP\nfmb/E2/ZJx6pOHq0PZ6O99/3u+0SjR2b+vSr2Zg710Jz+3Z7vYIC23+zapV1d3z0ke3Tadcu9fni\nU0m3eLvsMvtMN3hFpVTldWN+0EK6O559VnXiRPfzveIK1QkTVPfaS/Wbb+y+6mrVU0+1x7xNp6ee\nynxTcvFi29T/9tvajy1YYJt1U6ZY98rGjTb/2bP9abZts/t69lT9zW9UL7xQ9TvfUf3iC9VevdJr\ny/Tpqjt3pt/mulRV+bevvdY2v1VtM3L8eGvnc8+pPvaY3T7+eNv8vOAC+3vHDtt8PeII+3vnTtXH\nH7f3MXy46vLljW9jrm3ebF1ZQSgvVy0pqXnfzp22fp5xhi3b2bOtu6VnT9Vly2waQPXEE9N7jaIi\nmz4ZoNq+fePa77n8cn9d8bqCundX/c//dDP/hixbZuukdZOk7u5gSGfpySdVzzrL/Xx371b9/vft\nP5MYRFu3qh59tPX/VVb6/bAffODutV9/3V9RvT7GSZNUr7zSgn3tWutHmzFDdeBAe/zoo+25H3+s\n2q+frdxbtqj+5S+15+/1Yb/5ZuZtKy210OnRQ/UXv6j54b34YtUHH6w5/ezZ1g//gx+o/vSn1q6T\nT1Y94QR77nnnqR50kOrbb/v7AVRVb7vNHj/jDPtfhNnMmfal09S2b1c95hjVrl1VTzlF9ZFHVPv2\nVV240JZtr15WBDTk5pv9L9BE3jraWN4X9pVX+vN8+23Vo45SffHFxs8/Uwxpx/76V9Xzzw9m3rt2\n+ZVHopISq0LGjbOV/667gnntd97xA7pPH7v9ox+pzp2ruv/+/rTz5lkFnvjcSZP8FX7mzJrzXrbM\n7v/zn/37IhF/x+O6dapffmnv88sv7UvK+zCffbY/X+/n1lvtsYkTVZ95pvZ7mTPHwuHJJ+2L7eKL\n7Xnz51sIT55s1XWysjLbmhkzxtpRXp7xYkxbaalVUZ7PP7etpsZau9ZdmGWjrMz+t97/+q9/9XdY\n3nqrav/+qu++m/q5a9bY//7qq+053vLZvNmKBEC1VSvVG27I/n9TUeEvn+pq1XvuUX3ttezm5QpD\n2rFHHrHN/aZWVeVvBiYGpGvbttlrVVXZh+Oii2wTs6EAicVsZZ85U3XoUBu18vLLtpc7GrV2t2tn\nXzBbt9rfBQVWxZ9+es0QHjnSfhcXW2AmPvbjH9sogwkTbJP+rbdSt6ey0u8iisVUP/00vfdfVaX6\n299aGLRvbyNp3nsv3aWXvsmT7f148/7d77TWCBXPhRfW7H5StVEE69bZsvzzn63rydtiyWVIpzJ/\nvuobb9jtF1+0wG3TRvW001RXrPCnGz7cRt5MmmRFQZ8+qtOmqT7/vP+evv7aiqR+/VTvvde+FBJt\n3WpbUZWVqdvyxRf5t3wY0g5Nn6563HEWFC3JkiWpK/y67N5tfcJe2HbpYhXxG2/YB6hVK9Vhw6zS\nPfRQ+/viVWjqAAAJUElEQVSmm2wztLxc9dJLreLt3t2e//Of276ANWts/rt22Yd31Ci//961rVut\nSj/uOPtymTLFKvTEYVWZ+PLLmkPaJkzww+LZZ1Uvu8xuX3ih6quvWhfBrFnWXw5YOxYt8l+/d2/V\nAQP8vnevCwqw5bdoUaMXQWC2bbMv7ssuU917b9Uzz7StKsAq7c6dbV2ZN8+C3OumeuABfx5z5vjr\nyBVX2Jd1aal9CXjTz51ry2v+fNXrr7fnvfOO6iGH2BDNfFFXSIs95o6IqOt55htvb/kvf2lHclHD\nysqAp56y0RfDh9t9q1fbsuzf3+Jl166aw70827fbFddHjGjaNidbt87O0/LYYzYC59RT7cQ5vXrZ\niIOePW06b/3Yvt2GcM2ZY4fm3323PTZ5ss3jmWfsUOHp0230wGOP2XlWnn3WTrL/r38BCxf6J446\n8EA7xPuf/7QDiL73PRuRAtiBFMOGAZdfbkM2hw+3gyTCYvNmWx6vv24HHx15pI2quOsu/yCSNWts\nJEmq0RMrV9qRtE89ZSN0WrWy4W0bNwL//reNMPFiaexYm2biRDtjXb4QEaiq1Lo/k0AVkZMB3AMb\nuvewqt6ZYppmHdLV1f7wo7vvTn84ETUvy5fb2PXly+3LxjvEt00bC+9Onex8Gf362TCrxYvtaLMX\nX7QvogMPtNAfOtSGXXpBFIvVPFw7FrN1rrLSgsc7RHnFCgv0XbtsSNwHH9hZFVMdFdfSVFXZOdVH\njbJlrWrDVT/+2M5d8uGHVjSMG2fn3MgXjQ5pESkA8DmAcQDWAZgL4FxVXZY0XWhD+sknrdI5/ng7\nLSNQ85SOySeXnzHDDkQJUjQaRSQSCfZFWpCglmdlpY2v3bTJDoPevNmqvu3b7UoeF15oB/307w8c\nfbRVyqp2O6y4brpVV0hncH0AjAKwXFVXxWf4JIDTASyr91mNMHcucPLJwZ0OMTGAy8vtxESAHdhw\nxx22iTRkiG1atm9vH6xExx8fTLsS8YPgVlDLs7DQTrA1YEDNU1gmGjbMv13XNGHCdbNpZHLEYV8A\nied3+jp+X6MkFt3JBfhrr9mRd7NmRQFYpeJdALIub7xR86xusZgdDVdRYVXO9OnAL38ZxQUXAF26\n+IcNJ176xjuk2Dvp0YcfWqX0ySf+NM8+a4d8Jopmcjq5HHLdziDed0tdlmGaZxDC8t6bcnlmUkmn\nbcKE5BGtqX82brTj+r/zHfv79detH6lLFwvXefNsfpdeGsX++0fw9tv298knW7fE+vXWrzdsmB/w\n3k6DESPsse7dbcfBgw/aThgTRatWEey3n+1E6Nev5ukFH3zQLmH/zTd2Zrvx462fS9UOPW7Xzs7J\nmywslYXrdgbxvlvqsgzTPIMQlvfelMszkz7pYwAUqerJ8b9/ARsycmfSdOHskCYiyrHG7jhsBeAz\n2I7D9QA+BHCeqi6t94lERJS1tLs7VLVaRK4CMAv+EDwGNBFRgJwfzEJERO40OLpDRPqJyFsislhE\nPhGRqfH7u4rILBH5TEReE5EuCc+5UUSWi8hSETkpft9eIrJARObHf28Wkd8F99byk6vlGb//PBFZ\nJCILReQVEemWi/eUS46X5zki8nF8Pnfk4v3kUqbLUkS6xacvE5F7k+Z1ZHzd/FxE7kn1epSmVMeK\nJ/4A6A3g8PjtvWD90gcBuBPA/4vffwOA38ZvHwxgAawrZRCALxCv2JPmOw/AsQ29fnP7cbU8AbQC\nsBFA1/h0dwK4OdfvL8TLsxuAVQC6xad7FMDYXL+/PF+WHQCMBnAZgHuT5vUBgKPjt18BMD7X7y+s\nPw1W0qq6QVUXxm+XA1gKoB/sQJbH4pM9BmBi/PZpAJ5U1SpVXQlgOexAmD1E5AAA+6jq7IZev7lx\nuDy9vcCdREQAdIYdCdqiOFyeQwB8rqrexZ3eBHBGk7yJPJHpslTVHar6bwAVifMRkd4AOqnq3Phd\nM+Avf8pQRpfPEpFBAA4HMAdAL1XdCNg/F0D89DK1DnpZi9oHvZwDYGbmzW1eGrM8VbUKwJUAPoEd\nWDQMwMNN0vA81cj18wsAB4rIABFpDQuV/k3T8vyT5rKsS1/YOulxcuBbS5V2SIvIXgCeAXB1/Fs2\neY9jJnsgzwXwRAbTNzuNXZ7xIPkxgMNUtS8srG8Koq1h0NjlqapbYcvzKQDFAL4CUB1AU/Oe4886\nNVJaIR0PhGcA/FVVX4jfvVFEesUf7w1gU/z+tahZgfSL3+fNawSAVqq6oJFtDy1Hy/Nw2MFEK+P3\nPwXguwE3PS+5Wj9V9WVVPUZVj4WdTOzzpmh/PslwWdal3gygzKRbST8CYImqTku470UAF8VvXwjg\nhYT7zxWRNiIyGMD+sANfPOehhVfRcLM81wI4WES864WfCOtDbImcrJ8isk/8d1dYV1IenW24yWSy\nLBPtOVIu3iWyTURGxfeXTK7jOZSOhvYsAjgWttm3ELZXfD6Ak2F7w9+A7QGeBWDvhOfcCOvjWwrg\npKT5fQHggFzvMc3Vj8vlCdurviQ+rxcQH+nRkn4cL8+/A1gM4FMAZ+X6vYVkWX4FYAuAUgCrARwU\nv38krAtuOYBpuX5vYf7hwSxERHkso9EdRETUtBjSRER5jCFNRJTHGNJERHmMIU1ElMcY0kREeYwh\nTaEmItXx099+Gj8F7nXxAyjqe85AETmvqdpI1BgMaQq77ap6pKoeAjvq8hQAtzTwnMEAzg+8ZUQO\nMKSp2VDVLbCjMK8C9lTM74jIvPjPMfFJ7wBwXLwCv1pECkTk/4vIB/ELKEzJ1XsgSsYjDinURKRU\nVTsn3VcC4EAAZQBiqrpbRPYH8ISqHi0iYwD8TFVPi08/BXZ+89tFpA2A2QDOVNVVTftuiGpL+0K0\nRCHi9Um3AXCfiBwOOyfF0DqmPwnAoSJyVvzvzvFpGdKUcwxpalZEZAiAKlXdLCK3ANigqiNEpBWA\nnXU9DcBPVfX1JmsoUZrYJ01ht2ckR/xUo38C8If4XV0ArI/fngy7LiRg3SCdEubxGoAr4+dShogM\nFZH2QTaaKF2spCns2onIfFjXRiWAGar6+/hjfwTwrIhMBvAvANvj9y8CEBORBQD+oqrT4peLmh8f\nvrcJvCYf5QnuOCQiymPs7iAiymMMaSKiPMaQJiLKYwxpIqI8xpAmIspjDGkiojzGkCYiymMMaSKi\nPPZ/4MwoKKGQGS8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0xe89af98>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "flows.plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** If your observed data had gaps during your simulation period, this would be a good point to establish the overlapping period in order to inform the simulation/calibration dates.\n",
    "\n",
    "In our case, the data aligns with the simulation, so the simulation dates we want are just the start and end of the time series:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Timestamp('2007-01-01 00:00:00'), Timestamp('2010-12-31 00:00:00'))"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "start,end = flows.index[[0,-1]]\n",
    "start,end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This (synthetic) observed flow sequence relates to the (synthetic) gauge towards the bottom of the system. What was it called?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['G123456A',\n",
       " 'Generated node name #1',\n",
       " 'Generated node name #2',\n",
       " 'Generated node name #3',\n",
       " 'Generated node name #4',\n",
       " 'Generated node name #5',\n",
       " 'Generated node name #6',\n",
       " 'Generated node name #7',\n",
       " 'Generated node name #8',\n",
       " 'Generated node name #9',\n",
       " 'Generated node name #10',\n",
       " 'Generated node name #11',\n",
       " 'Generated node name #12',\n",
       " 'Generated node name #13',\n",
       " 'Generated node name #14',\n",
       " 'Generated node name #15',\n",
       " 'Generated node name #16',\n",
       " 'Generated node name #17',\n",
       " 'Generated node name #18',\n",
       " 'Generated node name #19',\n",
       " 'Generated node name #20',\n",
       " 'Generated node name #21',\n",
       " 'Generated node name #22',\n",
       " 'Generated node name #23',\n",
       " 'Generated node name #24',\n",
       " 'Generated node name #25',\n",
       " 'Generated node name #26',\n",
       " 'Generated node name #27',\n",
       " 'Generated node name #28']"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "network = v.network()\n",
    "nodes = network['features'].find_by_feature_type('node')\n",
    "nodes._all_values('name')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Aaah, we want `'G123456A'`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "calibration_node = 'G123456A'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can tell PEST about the observations and the comparison we want.\n",
    "\n",
    "We need to tell PEST how to load the observed data - and the pandas command we used to do a test load will help:\n",
    "\n",
    "```python\n",
    "pd.read_csv('SyntheticObservedFlow.csv',parse_dates=True,dayfirst=True,index_col=0)\n",
    "# will become\n",
    "calibration.observations.data.read_csv('SyntheticObservedFlow.csv',parse_dates=True,dayfirst=True,index_col=0)\n",
    "```\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "calibration.observations.data.read_csv('SyntheticObservedFlow.csv',parse_dates=True,dayfirst=True,index_col=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And we can set up the comparison"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "comparison={'NetworkElement':calibration_node,'RecordingVariable':'Downstream Flow Volume'}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "veneer-py configures the observation based on the column name in the observed flow file (so that you can have multiple comparisons from different columns and files)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['Flow'], dtype='object')"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "flows.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We also need to reference a stats function. You can write your own (but you'll need to store it in a `.py` file) or you can access one from `veneer.stats`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from veneer import stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['PBIAS',\n",
       " '__builtins__',\n",
       " '__cached__',\n",
       " '__doc__',\n",
       " '__file__',\n",
       " '__loader__',\n",
       " '__name__',\n",
       " '__package__',\n",
       " '__spec__',\n",
       " 'intersect',\n",
       " 'ioad',\n",
       " 'nse',\n",
       " 'ppmc',\n",
       " 'rae',\n",
       " 'rsr']"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dir(stats)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function nse in module veneer.stats:\n",
      "\n",
      "nse(obs, pred)\n",
      "    Nash-Sutcliffe Efficiency\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(stats.nse)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "calibration.observations.compare('Flow',comparison,stat=stats.nse,aggregation='daily')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to do one more thing: We need to make sure that each of our _n_ servers is configured to record the output we require.\n",
    "\n",
    "We'll need to loop over each of our veneer clients and configure recording. We have `comparison` which describes what we want to record. We can disable all other outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Configuring recording for server on port 9950\n",
      "Configuring recording for server on port 9951\n",
      "Configuring recording for server on port 9952\n",
      "Configuring recording for server on port 9953\n",
      "Configuring recording for server on port 9954\n",
      "Configuring recording for server on port 9955\n",
      "Configuring recording for server on port 9956\n",
      "Configuring recording for server on port 9957\n",
      "Configuring recording for server on port 9958\n",
      "Configuring recording for server on port 9959\n",
      "Configuring recording for server on port 9960\n",
      "Configuring recording for server on port 9961\n",
      "Configuring recording for server on port 9962\n",
      "Configuring recording for server on port 9963\n",
      "Configuring recording for server on port 9964\n",
      "Configuring recording for server on port 9965\n",
      "Configuring recording for server on port 9966\n",
      "Configuring recording for server on port 9967\n",
      "Configuring recording for server on port 9968\n",
      "Configuring recording for server on port 9969\n"
     ]
    }
   ],
   "source": [
    "for v in vs:\n",
    "    veneer.log('Configuring recording for server on port %d'%v.port)\n",
    "    v.configure_recording(enable=[comparison],disable=[{}])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we look at the content of the PEST config files now, we'll see more details filled in:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ptf $\n",
      "from veneer.pest_runtime import *\n",
      "from veneer import Veneer\n",
      "from veneer.stats import * \n",
      "import pandas as pd\n",
      "from veneer import general\n",
      "\n",
      "general.PRINT_URLS=False\n",
      "veneer_port=find_port()\n",
      "v = Veneer(port=veneer_port)\n",
      "v.model.catchment.runoff.set_param_values('x1',$x1$)\n",
      "v.model.catchment.runoff.set_param_values('x2',$x2$)\n",
      "v.model.catchment.runoff.set_param_values('x4',$x4$)\n",
      "v.model.catchment.runoff.set_param_values('x3',$x3$)\n",
      "# Run Model\n",
      "v.drop_all_runs()\n",
      "v.run_model(**{})\n",
      "\n",
      "observed_ts={}\n",
      "pest_observations=[]\n",
      "\n",
      "# Get results\n",
      "run_results = v.retrieve_run()\n",
      "\n",
      "# Compute stats\n",
      "observed_ts.update(pd.read_csv('SyntheticObservedFlow.csv',index_col=0,parse_dates=True,dayfirst=True).dropna(how='all').to_dict('series'))\n",
      "mod_ts = v.retrieve_multiple_time_series(run_data=run_results,criteria={'NetworkElement': 'G123456A', 'RecordingVariable': 'Downstream Flow Volume'},timestep=\"daily\")\n",
      "print(mod_ts.columns)\n",
      "assert(len(mod_ts.columns==0))\n",
      "mod_ts = mod_ts[mod_ts.columns[0]]\n",
      "obs_ts = observed_ts[\"Flow\"].dropna()\n",
      "pest_observations.append((\"Flow\",1.000000-nse(obs_ts,mod_ts)))\n",
      "\n",
      "# Write summary results\n",
      "print(pest_observations)\n",
      "write_outputs(pest_observations,'_CalibrationCase_output.txt')\n",
      "write_outputs(pest_observations,'__outputs_to_keep.txt')\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(calibration.ptf_text())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pif $\n",
      "$Flow$ !Flow!\n"
     ]
    }
   ],
   "source": [
    "print(calibration.pif_text())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "pcf\n",
      "* control data\n",
      "restart estimation\n",
      "4 1 1 0 1  \n",
      "1 1 single point    \n",
      "10.0 2.0 0.3 0.03 8   \n",
      "10.0 10.0 0.001   \n",
      "0.1     \n",
      "50 0.005 4 4 0.005 4   \n",
      "1 1 1       \n",
      "* singular value decomposition\n",
      "1\n",
      "4 5e-07\n",
      "0\n",
      "* parameter groups\n",
      "default_pg relative 0.001 0.0001 switch 1.5 parabolic   \n",
      "* parameter data\n",
      "x1 none factor 300.0 100.0 500.0 default_pg 1.0 0.0 1\n",
      "x2 none factor 3.0 1.0 5.0 default_pg 1.0 0.0 1\n",
      "x4 none factor 1.75 0.5 3.0 default_pg 1.0 0.0 1\n",
      "x3 none factor 100.5 1.0 200.0 default_pg 1.0 0.0 1 \n",
      "* observation groups\n",
      "default_og  \n",
      "* observation data\n",
      "Flow 0 1.0 default_og\n",
      "* model command line\n",
      "python _run_CalibrationCase.py\n",
      "* model input/output\n",
      "_run_CalibrationCase.tpl _run_CalibrationCase.py\n",
      "_CalibrationCase_output.ins _CalibrationCase_output.txt\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(calibration.pcf_text())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "prf\n",
      "20 0 0.5 1 0 run_slow_fac=1.5\n",
      "Slave_9950 .\\Slave_9950\\\n",
      "Slave_9951 .\\Slave_9951\\\n",
      "Slave_9952 .\\Slave_9952\\\n",
      "Slave_9953 .\\Slave_9953\\\n",
      "Slave_9954 .\\Slave_9954\\\n",
      "Slave_9955 .\\Slave_9955\\\n",
      "Slave_9956 .\\Slave_9956\\\n",
      "Slave_9957 .\\Slave_9957\\\n",
      "Slave_9958 .\\Slave_9958\\\n",
      "Slave_9959 .\\Slave_9959\\\n",
      "Slave_9960 .\\Slave_9960\\\n",
      "Slave_9961 .\\Slave_9961\\\n",
      "Slave_9962 .\\Slave_9962\\\n",
      "Slave_9963 .\\Slave_9963\\\n",
      "Slave_9964 .\\Slave_9964\\\n",
      "Slave_9965 .\\Slave_9965\\\n",
      "Slave_9966 .\\Slave_9966\\\n",
      "Slave_9967 .\\Slave_9967\\\n",
      "Slave_9968 .\\Slave_9968\\\n",
      "Slave_9969 .\\Slave_9969\\\n",
      "1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(calibration.prf_text())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running PEST\n",
    "\n",
    "We can now invoke the PEST run.\n",
    "\n",
    "When you call `calibration.run()`, PEST will start, and, in this case, it will start parallel PEST mode with _n_ workers. However all output of PEST will be in files, and in the command prompt window from which you started the Jupyter notebook. You will need to look in these places for progress of the calibration.\n",
    "\n",
    "When PEST finishes, you'll get the calibrated parameters back as well as details of the PEST run. The PEST manual covers the outputs of a PEST run in much detail"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** PEST needs to be in your Windows path. If its not, the `run()` command won't work. You can temporarily add PEST to your path from within the notebook:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "pest_path='C:\\\\PEST'\n",
    "import os\n",
    "os.environ['PATH'] = os.environ['PATH']+';'+pest_path"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "results = calibration.run()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>offset</th>\n",
       "      <th>scale</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>parameter</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>x1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>400.128906</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>x2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.198749</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>x4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.199333</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>x3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>54.981930</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "           offset  scale       value\n",
       "parameter                           \n",
       "x1            0.0    1.0  400.128906\n",
       "x2            0.0    1.0    1.198749\n",
       "x4            0.0    1.0    2.199333\n",
       "x3            0.0    1.0   54.981930"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#results = calibration.get_results()\n",
    "results['parameters']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "kill_all_now(processes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## While you're waiting...\n",
    "\n",
    "Now would be a good time to start sketching out an exercise of your own to tackle with veneer-py!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}