doc/training/7_Parallel_Processing_and_Veneer_Command_Line.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Session 7 - Parallel Processing and the Veneer command line\n",
    "\n",
    "This session looks at options for parallel processing with Veneer - that is, by running multiple copies of Source, each with a Veneer server running, and giving instructions to each running copy in parallel.\n",
    "\n",
    "You can establish multiple copies of Source/Veneer by running multiple copies of the Source application, loading a project and starting the Web Server Monitoring window on each one. Alternatively, you can use the Vener Command Line, which presents the same interface to Python and other systems, without the overheads of the user interface.\n",
    "\n",
    "This session shows how you can run the Veneer command line and use it from Python as you would the main Source application. \n",
    "\n",
    "## Overview\n",
    "\n",
    "- Launching multiple copies of Veneer command line using veneer-py\n",
    "- Running simulations in parallel\n",
    "\n",
    "## Which Model?\n",
    "\n",
    "**Note:** This session uses `ExampleProject/RiverModel1.rsproj`. You are welcome to work with your own model instead, however you will need to change the notebook text at certain points to reflect the names of nodes, links and functions in your model file.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Veneer Command Line\n",
    "\n",
    "The Veneer command line is a standalone executable program that runs the Source engine and exposes the Veneer network interface, without the main Source user interface. This means that all the veneer-py functionality able to be used whether you are running the Source application or the Veneer command line.\n",
    "\n",
    "### Setting up the command line\n",
    "\n",
    "The Veneer Command Line is distributed with Veneer, although the setup process is a little different to the regular Veneer. The Veneer Command Line is a standalone, executable program (specifically `FlowMatters.Source.VeneerCmd.exe`) and can be found in the directory where you installed (probably unzipped) Veneer.\n",
    "\n",
    "Now, where the Veneer plugin DLL can be installed and used from any directory, the Veneer Command Line needs access to all of the main libraries (DLLs) supplied with Source - and so the Veneer Command Line and the Source DLLs must reside in the same directory.\n",
    "\n",
    "There are two options. You can either\n",
    "\n",
    "1. copy the program and the other files supplied with Veneer, into your main Source installation directory, or\n",
    "2. copy ALL of the files from the main Source directory into a common directory with Veneer.\n",
    "\n",
    "Once you've done so, you should be able to run the Veneer command line. You can launch the Veneer command line directory from a Windows command prompt. Alternatively, you can start one or more copies directly from Python using `veneer.manage.start` (described below).\n",
    "\n",
    "Because it is a common requirement for everyone to co-locate the Veneer Command Line with the files from the Source software, veneer-py includes a `create_command_line` function that performs the copy for you:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from veneer.manage import create_command_line"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function create_command_line in module veneer.manage:\n",
      "\n",
      "create_command_line(veneer_path, source_version='4.1.1', source_path='C:\\\\Program Files\\\\eWater', dest=None)\n",
      "    Copy all Veneer related files and all files from the relevant Source distribution to a third directory,\n",
      "    for use as the veneer command line.\n",
      "    \n",
      "    veneer_path: Directory containing the Veneer files\n",
      "    \n",
      "    source_version: Version of Source to locate and copy\n",
      "    \n",
      "    source_path: Base installation directory for eWater Source\n",
      "    \n",
      "    dest: Destination to copy Source and Veneer to. If not provided, a temporary directory will be created.\n",
      "    \n",
      "    Returns: Full path to FlowMatters.Source.VeneerCmd.exe for use with start()\n",
      "    \n",
      "    Note: It is your responsibility to delete the copy of Source and Veneer from the destination directory\n",
      "    when you are finished with it! (Even if a temporary directory is used!)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(create_command_line)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'E:\\\\temp\\\\veneer_cmd\\\\FlowMatters.Source.VeneerCmd.exe'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "veneer_install = 'D:\\\\src\\\\projects\\\\Veneer\\\\Compiled\\\\Source 4.1.1.4484 (public version)'\n",
    "source_version = '4.1.1'\n",
    "cmd_directory = 'E:\\\\temp\\\\veneer_cmd'\n",
    "veneer_cmd = create_command_line(veneer_install,source_version,dest=cmd_directory)\n",
    "veneer_cmd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Starting the Command Line\n",
    "\n",
    "You can run `FlowMatters.Source.VeneerCmd.exe` program from a windows command prompt, but throughout these tutorials, we will use veneer-py functions for starting and stopping the program.\n",
    "\n",
    "Specifically, we will use `start` to start one or more copies of the program and `kill_all_now` to shutdown the program. (Alternatively, they will shutdown when the Jupyter notebook is shutdown)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from veneer.manage import start,kill_all_now"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Help on function start in module veneer.manage:\n",
      "\n",
      "start(project_fn, n_instances=1, ports=9876, debug=False, remote=True, script=True, veneer_exe=None)\n",
      "    Start one or more copies of the Veneer command line progeram with a given project file\n",
      "    \n",
      "    Parameters:\n",
      "    \n",
      "    - project_fn - Path to a Source project file (.rsproj)\n",
      "    \n",
      "    - n_instances - Number of copies of the Veneer command line to start (default: 1)\n",
      "    \n",
      "    - ports - A single port number, indicating the port number of the first copy of the Veneer command line,\n",
      "              OR a list of ports, in which case len(ports)==n_instances  (default: 9876)\n",
      "    \n",
      "    - debug - Set to True to echo all output from Veneer Command Line during startup\n",
      "    \n",
      "    - remote - Allow remote connections (requires registration)\n",
      "    \n",
      "    - script - Allow IronPython scripts\n",
      "    \n",
      "    - veneer_exe - Optional (but often required) path to the Veneer Command Line. If not provided,\n",
      "                   veneer-py will attempt to identify the version of Veneer Command Line to invoke.\n",
      "                   If there is a source_version.txt file in the same directory as the project file,\n",
      "                   this text file will be consulted to identify the version of Source.\n",
      "    \n",
      "    returns processes, ports\n",
      "       processes - list of process objects that can be used to terminate the servers\n",
      "       ports - the port numbers used for each copy of the server\n",
      "\n"
     ]
    }
   ],
   "source": [
    "help(start)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The main things you need, in order to call `start` are a Source project file (a path to the `.rsproj` file) and a path to the Veneer command line exe. (The latter we saved to variable `veneer_cmd` on calling `create_command_line`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "project='ExampleProject/RiverModel1.rsproj'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now start up a number of Veneer command line 'servers'.\n",
    "\n",
    "We'll specify how many we want using `num_copies` - Its a good idea to set this based on the number of CPU cores available.\n",
    "\n",
    "We also set `first_port` - Which is used for the first server. This number is incremented by one for each extra server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9990 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\RiverModel1.rsproj\"\n",
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9991 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\RiverModel1.rsproj\"\n",
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9992 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\RiverModel1.rsproj\"\n",
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9993 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\RiverModel1.rsproj\"\n",
      "[0] Loading plugins\n",
      "\n",
      "[1] Loading plugins\n",
      "\n",
      "[2] Loading plugins\n",
      "\n",
      "[3] Loading plugins\n",
      "\n",
      "[3] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[3] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[3] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[3] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[3] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[3] Plugins loaded (5/5)\n",
      "\n",
      "[0] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[0] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[0] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[0] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[0] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[0] Plugins loaded (5/5)\n",
      "\n",
      "[1] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[1] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[1] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[1] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[1] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[1] Plugins loaded (5/5)\n",
      "\n",
      "[2] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[2] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[2] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[2] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[2] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[2] Plugins loaded (5/5)\n",
      "\n",
      "[3] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\RiverModel1.rsproj\n",
      "\n",
      "[0] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\RiverModel1.rsproj\n",
      "\n",
      "[1] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\RiverModel1.rsproj\n",
      "\n",
      "[2] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\RiverModel1.rsproj\n",
      "\n",
      "ERROR[3] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "ERROR[0] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "ERROR[1] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "ERROR[2] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "[0] Loading project\n",
      "\n",
      "[2] Loading project\n",
      "\n",
      "[1] Loading project\n",
      "\n",
      "[3] Loading project\n",
      "\n",
      "[0] Project Loaded\n",
      "\n",
      "[0] New Project\n",
      "\n",
      "[0] Scenario 1\n",
      "\n",
      "[2] Project Loaded\n",
      "\n",
      "[2] New Project\n",
      "\n",
      "[2] Scenario 1\n",
      "\n",
      "[0] [3:04:30 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[0] [3:04:30 PM] Started Source RESTful Service on port:9990\n",
      "\n",
      "Server 0 on port 9990 is ready\n",
      "[0] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n",
      "[2] [3:04:30 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[2] [3:04:30 PM] Started Source RESTful Service on port:9992\n",
      "\n",
      "Server 2 on port 9992 is ready\n",
      "[2] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n",
      "[1] Project Loaded\n",
      "\n",
      "[1] New Project\n",
      "\n",
      "[1] Scenario 1\n",
      "\n",
      "[1] [3:04:31 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[1] [3:04:31 PM] Started Source RESTful Service on port:9991\n",
      "\n",
      "Server 1 on port 9991 is ready\n",
      "[1] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n",
      "[3] Project Loaded\n",
      "\n",
      "[3] New Project\n",
      "\n",
      "[3] Scenario 1\n",
      "\n",
      "[3] [3:04:31 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[3] [3:04:31 PM] Started Source RESTful Service on port:9993\n",
      "\n",
      "Server 3 on port 9993 is ready\n",
      "[3] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n"
     ]
    }
   ],
   "source": [
    "num_copies=4\n",
    "first_port=9990\n",
    "processes, ports = start(project,n_instances=num_copies,ports=first_port,debug=True,remote=False,veneer_exe=veneer_cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should see a number of lines along the lines of `[3] Server started. Ctrl-C to exit...` indicating that the servers have started.\n",
    "\n",
    "These servers will now run until your current python session ends. (To make that happen, without closing the notebook, use the `Kernel|Restart` menu option in Jupyter)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parallel Simulations\n",
    "\n",
    "You can now work with each of these Veneer servers in the same way that you worked with a single server in the earlier sessions.\n",
    "\n",
    "You will need an instance of the Veneer client object - one for each instance.\n",
    "\n",
    "Here, we'll create a list of Veneer clients, each connected to a different instance based on the port number"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import veneer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[9990, 9991, 9992, 9993]"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ports # Saved when we called start()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "vs = [veneer.Veneer(port=p) for p in ports]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can now ask one of these servers to run a model for you:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Default Link #4:Downstream Flow Volume</th>\n",
       "      <th>Default Link #5:Downstream Flow Volume</th>\n",
       "      <th>End Of System:Downstream Flow Volume</th>\n",
       "      <th>MFR:Downstream Flow Volume</th>\n",
       "      <th>SP:Downstream Flow Volume</th>\n",
       "      <th>SR2:Downstream Flow Volume</th>\n",
       "      <th>SR3:Downstream Flow Volume</th>\n",
       "      <th>Storage:Downstream Flow Volume</th>\n",
       "      <th>Ungauged Inflow:Downstream Flow Volume</th>\n",
       "      <th>Water User:Downstream Flow Volume</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1901-01-01</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>514.15</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>514.15</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1901-01-02</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>483.35</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>483.35</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1901-01-03</th>\n",
       "      <td>6349.027111</td>\n",
       "      <td>1815.10</td>\n",
       "      <td>6349.027111</td>\n",
       "      <td>6349.027111</td>\n",
       "      <td>6349.027111</td>\n",
       "      <td>6349.027111</td>\n",
       "      <td>6349.027111</td>\n",
       "      <td>6349.027111</td>\n",
       "      <td>1815.10</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1901-01-04</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>2857.05</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2857.05</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1901-01-05</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>2763.25</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2763.25</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1901-01-06</th>\n",
       "      <td>316601.314118</td>\n",
       "      <td>11478.60</td>\n",
       "      <td>316601.314118</td>\n",
       "      <td>316601.314118</td>\n",
       "      <td>316601.314118</td>\n",
       "      <td>316601.314118</td>\n",
       "      <td>316601.314118</td>\n",
       "      <td>316601.314118</td>\n",
       "      <td>11478.60</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1901-01-07</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>8350.30</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>8350.30</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1901-01-08</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>5412.40</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>5412.40</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1901-01-09</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>3761.10</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>3761.10</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1901-01-10</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>2825.20</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>2825.20</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            Default Link #4:Downstream Flow Volume  \\\n",
       "1901-01-01                                0.000000   \n",
       "1901-01-02                                0.000000   \n",
       "1901-01-03                             6349.027111   \n",
       "1901-01-04                                0.000000   \n",
       "1901-01-05                                0.000000   \n",
       "1901-01-06                           316601.314118   \n",
       "1901-01-07                                0.000000   \n",
       "1901-01-08                                0.000000   \n",
       "1901-01-09                                0.000000   \n",
       "1901-01-10                                0.000000   \n",
       "\n",
       "            Default Link #5:Downstream Flow Volume  \\\n",
       "1901-01-01                                  514.15   \n",
       "1901-01-02                                  483.35   \n",
       "1901-01-03                                 1815.10   \n",
       "1901-01-04                                 2857.05   \n",
       "1901-01-05                                 2763.25   \n",
       "1901-01-06                                11478.60   \n",
       "1901-01-07                                 8350.30   \n",
       "1901-01-08                                 5412.40   \n",
       "1901-01-09                                 3761.10   \n",
       "1901-01-10                                 2825.20   \n",
       "\n",
       "            End Of System:Downstream Flow Volume  MFR:Downstream Flow Volume  \\\n",
       "1901-01-01                              0.000000                    0.000000   \n",
       "1901-01-02                              0.000000                    0.000000   \n",
       "1901-01-03                           6349.027111                 6349.027111   \n",
       "1901-01-04                              0.000000                    0.000000   \n",
       "1901-01-05                              0.000000                    0.000000   \n",
       "1901-01-06                         316601.314118               316601.314118   \n",
       "1901-01-07                              0.000000                    0.000000   \n",
       "1901-01-08                              0.000000                    0.000000   \n",
       "1901-01-09                              0.000000                    0.000000   \n",
       "1901-01-10                              0.000000                    0.000000   \n",
       "\n",
       "            SP:Downstream Flow Volume  SR2:Downstream Flow Volume  \\\n",
       "1901-01-01                   0.000000                    0.000000   \n",
       "1901-01-02                   0.000000                    0.000000   \n",
       "1901-01-03                6349.027111                 6349.027111   \n",
       "1901-01-04                   0.000000                    0.000000   \n",
       "1901-01-05                   0.000000                    0.000000   \n",
       "1901-01-06              316601.314118               316601.314118   \n",
       "1901-01-07                   0.000000                    0.000000   \n",
       "1901-01-08                   0.000000                    0.000000   \n",
       "1901-01-09                   0.000000                    0.000000   \n",
       "1901-01-10                   0.000000                    0.000000   \n",
       "\n",
       "            SR3:Downstream Flow Volume  Storage:Downstream Flow Volume  \\\n",
       "1901-01-01                    0.000000                        0.000000   \n",
       "1901-01-02                    0.000000                        0.000000   \n",
       "1901-01-03                 6349.027111                     6349.027111   \n",
       "1901-01-04                    0.000000                        0.000000   \n",
       "1901-01-05                    0.000000                        0.000000   \n",
       "1901-01-06               316601.314118                   316601.314118   \n",
       "1901-01-07                    0.000000                        0.000000   \n",
       "1901-01-08                    0.000000                        0.000000   \n",
       "1901-01-09                    0.000000                        0.000000   \n",
       "1901-01-10                    0.000000                        0.000000   \n",
       "\n",
       "            Ungauged Inflow:Downstream Flow Volume  \\\n",
       "1901-01-01                                  514.15   \n",
       "1901-01-02                                  483.35   \n",
       "1901-01-03                                 1815.10   \n",
       "1901-01-04                                 2857.05   \n",
       "1901-01-05                                 2763.25   \n",
       "1901-01-06                                11478.60   \n",
       "1901-01-07                                 8350.30   \n",
       "1901-01-08                                 5412.40   \n",
       "1901-01-09                                 3761.10   \n",
       "1901-01-10                                 2825.20   \n",
       "\n",
       "            Water User:Downstream Flow Volume  \n",
       "1901-01-01                                  0  \n",
       "1901-01-02                                  0  \n",
       "1901-01-03                                  0  \n",
       "1901-01-04                                  0  \n",
       "1901-01-05                                  0  \n",
       "1901-01-06                                  0  \n",
       "1901-01-07                                  0  \n",
       "1901-01-08                                  0  \n",
       "1901-01-09                                  0  \n",
       "1901-01-10                                  0  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vs[0].run_model()\n",
    "vs[0].retrieve_multiple_time_series(criteria={'RecordingVariable':'Downstream Flow Volume'})[0:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You could run a model on each server using a for loop:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running on port 9990\n",
      "Running on port 9991\n",
      "Running on port 9992\n",
      "Running on port 9993\n",
      "All runs finished\n"
     ]
    }
   ],
   "source": [
    "for v in vs:\n",
    "    veneer.log('Running on port %d'%v.port)\n",
    "    v.run_model()\n",
    "print('All runs finished')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But that is a sequential run - One run won't start until the previous run has finished.\n",
    "\n",
    "The `run_async` option on `v.run_model` will trigger the run on the server and then allow Python to continue:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running on port 9990\n",
      "Running on port 9991\n",
      "Running on port 9992\n",
      "Running on port 9993\n",
      "All runs started... But when will they finish? And how will we know?\n"
     ]
    }
   ],
   "source": [
    "for v in vs:\n",
    "    veneer.log('Running on port %d'%v.port)\n",
    "    v.run_model(run_async=True)\n",
    "print('All runs started... But when will they finish? And how will we know?')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above code block flies through quickly, because it doesn't wait for the simulation to finish. But how will we know when the run has finished, so that we can continue our script?\n",
    "\n",
    "(We can look at Windows Task Manager to see the CPU load - but its not exactly a precise mechanism...)\n",
    "\n",
    "When run with `run_async=True`, `v.run_model` returns a HTTP connection object, that can be queried for the success code of the run. We can use this to block for a particular run to finish. Assuming we don't want to do anything else in our script until ALL runs are finished, this is a good approach:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running on port 9990\n",
      "Running on port 9991\n",
      "Running on port 9992\n",
      "Running on port 9993\n",
      "All runs started... Now we'll wait when until they finish\n",
      "Run finished on port 9990. Returned a HTTP 302 code\n",
      "Run finished on port 9991. Returned a HTTP 302 code\n",
      "Run finished on port 9992. Returned a HTTP 302 code\n",
      "Run finished on port 9993. Returned a HTTP 302 code\n"
     ]
    }
   ],
   "source": [
    "responses = []\n",
    "for v in vs:\n",
    "    veneer.log('Running on port %d'%v.port)\n",
    "    responses.append(v.run_model(run_async=True))\n",
    "\n",
    "veneer.log(\"All runs started... Now we'll wait when until they finish\")\n",
    "\n",
    "for r,v in zip(responses,vs):\n",
    "    code = r.getresponse().getcode()\n",
    "    veneer.log('Run finished on port %d. Returned a HTTP %d code'%(v.port,code))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can use this the `run_async` approach to run multiple, parallel simulations, from a notebook.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** We will shutdown those 4 copies of the server now as the next exercise will use a different model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "kill_all_now(processes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Running a batch simulation with parallel simulations...\n",
    "\n",
    "In Tutorial 5, we performed exponential sampling for an inflow scaling factor. We ran the model 50 times. Lets run that process again, using parallel processing.\n",
    "\n",
    "The code looked like this (combining a few notebook cells and removing some interim visualisation)\n",
    "\n",
    "```python\n",
    "import numpy as np\n",
    "NUMBER_OF_SIMULATIONS=50\n",
    "sampled_scaling_factors = np.random.exponential(size=NUMBER_OF_SIMULATIONS) \n",
    "sampled_scaling_factors\n",
    "\n",
    "spill_results=[]\n",
    "\n",
    "# Store our time series criteria in a variable to use it in configuring recording and retrieving results\n",
    "ts_match_criteria = {'NetworkElement':'Recreational Lake','RecordingVariable':'Spill Volume'}\n",
    "\n",
    "v.configure_recording(enable=[ts_match_criteria])\n",
    "\n",
    "for scaling_factor in sampled_scaling_factors:\n",
    "    veneer.log('Running for $InflowScaling=%f'%scaling_factor)\n",
    "\n",
    "    # We are running the multiple many times in this case - so lets drop any results we already have...\n",
    "    v.drop_all_runs()\n",
    "\n",
    "    # Set $InflowScaling to current scaling factor\n",
    "    v.update_function('$InflowScaling',scaling_factor)\n",
    "    v.run_model()\n",
    "    \n",
    "    # Retrieve the spill time series, as an annual sum, with the column named for the variable ('Spill Volume')\n",
    "    run_results = v.retrieve_multiple_time_series(criteria=ts_match_criteria,timestep='annual',name_fn=veneer.name_for_variable)\n",
    "    \n",
    "    # Store the mean spill volume and the scaling factor we used\n",
    "    spill_results.append({'ScalingFactor':scaling_factor,'SpillVolume':run_results['Spill Volume'].mean()})\n",
    "\n",
    "# Convert the results to a Data Frame\n",
    "spill_results_df = pd.DataFrame(spill_results)\n",
    "spill_results_df\n",
    "```\n",
    "\n",
    "Lets convert this to something that runs in parallel\n",
    "\n",
    "First, lets set up the servers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9990 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\RiverModel2.rsproj\"\n",
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9991 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\RiverModel2.rsproj\"\n",
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9992 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\RiverModel2.rsproj\"\n",
      "Starting E:\\temp\\veneer_cmd\\FlowMatters.Source.VeneerCmd.exe -p 9993 -s  \"D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\RiverModel2.rsproj\"\n",
      "[0] Loading plugins\n",
      "\n",
      "[1] Loading plugins\n",
      "\n",
      "[2] Loading plugins\n",
      "\n",
      "[3] Loading plugins\n",
      "\n",
      "[3] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[3] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[3] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[3] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[3] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[3] Plugins loaded (5/5)\n",
      "\n",
      "[0] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[0] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[0] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[0] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[0] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[0] Plugins loaded (5/5)\n",
      "\n",
      "[1] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[1] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[1] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[1] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[1] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[1] Plugins loaded (5/5)\n",
      "\n",
      "[2] Loaded D:\\src\\projects\\Veneer\\Compiled\\GBRSource\\FlowMatters.Source.Veneer.dll\n",
      "\n",
      "[2] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\Dynamic_SedNet.dll\n",
      "\n",
      "[2] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\GBR_DynSed_Extension.dll\n",
      "\n",
      "[2] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\CommunityPlugins\\ReefHydroCalModels.dll\n",
      "\n",
      "[2] Loaded D:\\src\\projects\\gbr\\Output\\Plugins\\DERMTools.dll\n",
      "\n",
      "[2] Plugins loaded (5/5)\n",
      "\n",
      "[0] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\RiverModel2.rsproj\n",
      "\n",
      "[1] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\RiverModel2.rsproj\n",
      "\n",
      "[2] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\RiverModel2.rsproj\n",
      "\n",
      "ERROR[3] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "[3] Opening project file: D:\\src\\projects\\veneer-py\\doc\\training\\ExampleProject\\RiverModel2.rsproj\n",
      "\n",
      "ERROR[0] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "ERROR[1] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "ERROR[2] log4net:ERROR Failed to find configuration section 'log4net' in the application's .config file. Check your .config file for the <log4net> and <configSections> elements. The configuration section should look like: <section name=\"log4net\" type=\"log4net.Config.Log4NetConfigurationSectionHandler,log4net\" />\n",
      "\n",
      "[1] Loading project\n",
      "\n",
      "[3] Loading project\n",
      "\n",
      "[0] Loading project\n",
      "\n",
      "[2] Loading project\n",
      "\n",
      "[1] Project Loaded\n",
      "\n",
      "[1] River\n",
      "\n",
      "[1] WaterBalance\n",
      "\n",
      "[1] [3:12:41 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[1] [3:12:41 PM] Started Source RESTful Service on port:9991\n",
      "\n",
      "Server 1 on port 9991 is ready\n",
      "[1] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n",
      "[3] Project Loaded\n",
      "\n",
      "[3] River\n",
      "\n",
      "[3] WaterBalance\n",
      "\n",
      "[3] [3:12:42 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[3] [3:12:42 PM] Started Source RESTful Service on port:9993\n",
      "\n",
      "Server 3 on port 9993 is ready\n",
      "[3] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n",
      "[0] Project Loaded\n",
      "\n",
      "[0] River\n",
      "\n",
      "[0] WaterBalance\n",
      "\n",
      "[2] Project Loaded\n",
      "\n",
      "[2] River\n",
      "\n",
      "[2] WaterBalance\n",
      "\n",
      "[0] [3:12:42 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[0] [3:12:42 PM] Started Source RESTful Service on port:9990\n",
      "\n",
      "Server 0 on port 9990 is ready\n",
      "[0] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n",
      "[2] [3:12:42 PM] Veneer, by Flow Matters: http://www.flowmatters.com.au\n",
      "\n",
      "[2] [3:12:42 PM] Started Source RESTful Service on port:9992\n",
      "\n",
      "Server 2 on port 9992 is ready\n",
      "[2] Server started. Ctrl-C to exit, or POST /shutdown command\n",
      "\n"
     ]
    }
   ],
   "source": [
    "project='ExampleProject/RiverModel2.rsproj'\n",
    "num_copies=4\n",
    "first_port=9990\n",
    "processes, ports = start(project,n_instances=num_copies,ports=first_port,debug=True,remote=False,veneer_exe=veneer_cmd)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need a veneer client object for each running server:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "vs = [veneer.Veneer(port=p) for p in ports]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The sampling process is the same as before...\n",
    "\n",
    "Except we'll use more samples **and make sure the number of samples is a multiple of the number of servers!**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0.21828055,  0.36392667,  0.07106982,  0.35537556,  0.49925756,\n",
       "        2.28313128,  2.44344286,  1.35232922,  0.17703713,  0.24183259,\n",
       "        0.00623003,  0.36957271,  1.00505532,  3.98117835,  0.21893658,\n",
       "        0.14258378,  0.41440569,  1.00467281,  0.23425366,  0.49765048,\n",
       "        2.48401778,  0.64162755,  0.03718622,  0.92356851,  1.00677409,\n",
       "        0.57647994,  0.98670485,  0.43695266,  1.70815562,  0.51411446,\n",
       "        3.41275018,  0.42179291,  0.67056163,  0.05074394,  0.39060761,\n",
       "        0.36341694,  0.11971328,  0.58866828,  0.63181065,  2.05749534,\n",
       "        0.08459884,  0.82368285,  0.30062613,  1.906684  ,  0.5379337 ,\n",
       "        0.8772698 ,  1.87724781,  0.81314884,  1.5228271 ,  0.26858821,\n",
       "        0.42374207,  1.94993642,  3.68202781,  1.04852028,  0.84775752,\n",
       "        0.36325295,  0.11497461,  0.0120431 ,  1.58120889,  0.78542681,\n",
       "        0.09562667,  0.42759238,  1.32524812,  1.17473154,  1.62626417,\n",
       "        3.2516898 ,  0.18496208,  1.68430385,  0.07515113,  1.58519883,\n",
       "        0.42267374,  2.33612248,  1.27921425,  0.92020851,  1.16684069,\n",
       "        3.00723003,  1.34674872,  0.42039373,  0.30650172,  1.07809742,\n",
       "        0.20273928,  0.36960488,  1.13041175,  0.49695429,  0.74731115,\n",
       "        1.85200001,  0.06864933,  0.18548627,  0.19659477,  1.11944598,\n",
       "        0.59404985,  0.82596846,  0.2110298 ,  1.72046115,  1.07363669,\n",
       "        2.3335147 ,  0.32605006,  1.53509757,  0.3124965 ,  0.47407891])"
      ]
     },
     "execution_count": 70,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "NUMBER_OF_SIMULATIONS=100\n",
    "sampled_scaling_factors = np.random.exponential(size=NUMBER_OF_SIMULATIONS) \n",
    "sampled_scaling_factors"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([ 34.,  21.,  18.,   8.,   8.,   4.,   2.,   1.,   2.,   2.]),\n",
       " array([ 0.00623003,  0.40372486,  0.80121969,  1.19871453,  1.59620936,\n",
       "         1.99370419,  2.39119902,  2.78869386,  3.18618869,  3.58368352,\n",
       "         3.98117835]),\n",
       " <a list of 10 Patch objects>)"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXUAAAEACAYAAABMEua6AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAEBBJREFUeJzt3W+IZXd9x/H3Z7O6bhoyWG12g6krpVStEFYfxErKuKKp\n0dJGpNiq4N+WUBqVCmoSKhskhfokxVJ8IImyaWurCDYb/yWR5EJiSRrMLtnomiptYiPO0GLcGlOD\ndr59MGdkMs7sPXfm3j03v7xfcMmZc86958NvNp9753fPPTdVhSSpDbuGDiBJmh5LXZIaYqlLUkMs\ndUlqiKUuSQ2x1CWpIWNLPcmeJPckOZbkRJLD3frDSR5Jcl93u3T2cSVJp5M+56knObuqHk9yFvA1\n4L3A64AfVdV1M84oSeqp1/RLVT3eLe4BdgNrzwSZRShJ0vb0KvUku5IcA5aA26rq3m7TFUmOJ7k+\nycLMUkqSeuk1/fLznZNzgc8D7wH+C/jvqqok1wLnV9W7ZxNTktTHRKUOkOTDwI/Xz6UnOQDcXFUX\nbrK/F5eRpG2oqomnuPuc/fLctamVJHuBS4BvJdm/brc3Ag+cJtjc3w4fPjx4BnOa0ZzmXLtt1+4e\n+5wPHEmyi9Ungc9U1ZeS3JjkILACPARcvu0UkqSpGFvqVXUCeNkm6982k0SSpG3zE6WdQ4cODR2h\nF3NOz1MhI5hz2p4qObdr4jdKJz5AUh/84FUzPcY4e/c+iw996APs3bt30ByS1FcSahtvlJ6RUoe/\nnOkxxnnWsz7BLbfcyOLi4qA5JKmv7ZZ6nzdKp+DqM3OYLezZ85VBjy9JZ4pz6pLUEEtdkhpiqUtS\nQyx1SWqIpS5JDbHUJakhlrokNcRSl6SGWOqS1BBLXZIaYqlLUkMsdUlqiKUuSQ2x1CWpIZa6JDXE\nUpekhljqktQQS12SGmKpS1JDxpZ6kj1J7klyLMmJJIe79c9OcmuSB5PckmRh9nElSaczttSr6gng\nVVX1UuAg8LokFwFXAl+tqhcCtwNXzTSpJGmsXtMvVfV4t7gH2A0UcBlwpFt/BHjD1NNJkibSq9ST\n7EpyDFgCbquqe4F9VbUMUFVLwHmziylJ6mN3n52qagV4aZJzgc8neQmrr9aftNvWj3DNuuVD3U2S\ntGY0GjEajXb8OKk6TRdvdofkw8DjwB8Dh6pqOcl+4I6qevEm+9dp+/4MWFhY5OjRa1lcXBw0hyT1\nlYSqyqT363P2y3PXzmxJshe4BDgJHAXe0e32duCmSQ8uSZquPtMv5wNHkuxi9UngM1X1pSR3A59N\n8i7gYeBNM8wpSephbKlX1QngZZus/wHwmlmEkiRtj58olaSGWOqS1BBLXZIaYqlLUkMsdUlqiKUu\nSQ2x1CWpIZa6JDXEUpekhljqktQQS12SGmKpS1JDLHVJaoilLkkNsdQlqSGWuiQ1xFKXpIZY6pLU\nEEtdkhpiqUtSQyx1SWqIpS5JDbHUJakhY0s9yQVJbk/yjSQnkrynW384ySNJ7utul84+riTpdHb3\n2OdnwPur6niSc4CvJ7mt23ZdVV03u3iSpEmMLfWqWgKWuuXHkpwEntdtzgyzSZImNNGcepIXAAeB\ne7pVVyQ5nuT6JAtTziZJmlCf6RcAuqmXzwHv616xfxz4SFVVkmuB64B3b37va9YtH+pukqQ1o9GI\n0Wi048dJVY3fKdkNfAH4clV9bJPtB4Cbq+rCTbYVjD/GLC0sLHL06LUsLi4OmkOS+kpCVU08xd13\n+uWTwDfXF3qS/eu2vxF4YNKDS5Kma+z0S5KLgbcCJ5IcY/Vl99XAW5IcBFaAh4DLZ5hTktRDn7Nf\nvgactcmmr0w/jiRpJ/xEqSQ1xFKXpIZY6pLUEEtdkhpiqUtSQyx1SWqIpS5JDbHUJakhlrokNcRS\nl6SGWOqS1BBLXZIaYqlLUkMsdUlqiKUuSQ2x1CWpIZa6JDXEUpekhljqktQQS12SGmKpS1JDUlWz\nPUBSMNtjjLOwsMiuXd/h0Ue/P2iOffsOsLT00KAZJD01JKGqMun9ds8izDxaLfRhn1yWlyf+/UjS\nRMZOvyS5IMntSb6R5ESS93brn53k1iQPJrklycLs40qSTqfPnPrPgPdX1UuAVwB/luRFwJXAV6vq\nhcDtwFWziylJ6mNsqVfVUlUd75YfA04CFwCXAUe63Y4Ab5hVSElSPxOd/ZLkBcBB4G5gX1Utw2rx\nA+dNO5wkaTK93yhNcg7wOeB9VfXY6lktT3KadyGvWbd8qLtJktaMRiNGo9GOH6fXKY1JdgNfAL5c\nVR/r1p0EDlXVcpL9wB1V9eJN7jsXpzSeOnUnQ+eAMOtTSCW1YbunNPadfvkk8M21Qu8cBd7RLb8d\nuGnSg0uSpmvs9EuSi4G3AieSHGP15e7VwEeBzyZ5F/Aw8KZZBpUkjTe21Kvqa8BZW2x+zXTjSJJ2\nwmu/SFJDLHVJaoilLkkNsdQlqSGWuiQ1xFKXpIZY6pLUEEtdkhrytPnmo/mwh2TYbz/yK/Wktlnq\nZ9QTDH1RMb9ST2qb0y+S1BBLXZIaYqlLUkMsdUlqiKUuSQ2x1CWpIZa6JDXEUpekhljqktQQS12S\nGmKpS1JDLHVJaoilLkkNGVvqSW5Ispzk/nXrDid5JMl93e3S2caUJPXR55X6p4DXbrL+uqp6WXf7\nypRzSZK2YWypV9VdwKObbPLC3JI0Z3Yyp35FkuNJrk+yMLVEkqRt2+43H30c+EhVVZJrgeuAd2+9\n+zXrlg91N0nSmtFoxGg02vHjpGr816slOQDcXFUXTrKt215Df4XbwsIip07dydA5Vmeshs/Q53cu\naVhJqKqJp7n7Tr+EdXPoSfav2/ZG4IFJDyxJmr6x0y9JPs3qfMlzknwXOAy8KslBYAV4CLh8hhkl\nST2NLfWqessmqz81gyySpB3yE6WS1BBLXZIaYqlLUkMsdUlqiKUuSQ2x1CWpIZa6JDXEUpekhljq\nktQQS12SGmKpS1JDLHVJaoilLkkNsdQlqSGWuiQ1xFKXpIZY6pLUEEtdkhpiqUtSQyx1SWqIpS5J\nDbHUJakhlrokNWRsqSe5IclykvvXrXt2kluTPJjkliQLs40pSeqjzyv1TwGv3bDuSuCrVfVC4Hbg\nqmkHkyRNbmypV9VdwKMbVl8GHOmWjwBvmHIuSdI2bHdO/byqWgaoqiXgvOlFkiRt1+4pPU6dfvM1\n65YPdTdJ0prRaMRoNNrx46RqTB8DSQ4AN1fVhd3PJ4FDVbWcZD9wR1W9eIv71tjOn7GFhUVOnbqT\noXNA5iJDn9+5pGEloaoy6f36Tr+ku605CryjW347cNOkB5YkTV+fUxo/DfwL8BtJvpvkncBfAZck\neRB4dfezJGlgY+fUq+otW2x6zZSzSJJ2yE+USlJDLHVJaoilLkkNsdQlqSGWuiQ1xFKXpIZY6pLU\nEEtdkhoyrQt66SljD8nEl5OYul27zmZl5fFBM+zbd4ClpYcGzSBNm6X+tPMEw19UDFZWhr+42fLy\n8E9u0rQ5/SJJDbHUJakhlrokNcRSl6SGWOqS1BBLXZIaYqlLUkMsdUlqiKUuSQ2x1CWpIZa6JDXE\nUpekhljqktSQHV2lMclDwClgBfhpVV00jVCSpO3Z6aV3V4BDVfXoNMJIknZmp9MvmcJjSJKmZKeF\nXMBtSe5N8ifTCCRJ2r6dTr9cXFXfT/IrrJb7yaq66xd3u2bd8qHuJklaMxqNGI1GO36cVE3nK8WS\nHAZ+VFXXbVhfQ39t2cLCIqdO3cnQOVZnq8ywah5yhGn9+5emLQlVNfF3Lm57+iXJ2UnO6ZZ/Cfgd\n4IHtPp4kaed2Mv2yD/j86itxdgP/UFW3TieWJGk7tl3qVfUfwMEpZpEk7ZCnI0pSQ3Z69ov0FLaH\nZOL3oaZq374DLC09NGgGtcVS19PYEwx9Bs7y8rBPKmqP0y+S1BBLXZIaYqlLUkMsdUlqiKUuSQ2x\n1CWpIZa6JDXEUpekhljqktQQS12SGmKpS1JDvPaLNCgvKrZm//4XsLz88KAZdu06m5WVxwfNsNPf\nh6UuDcqLiq1ZLfRhx2JlZfivWdzp78PpF0lqiKUuSQ2x1CWpIZa6JDXEN0qlp73hz8DR9Fjq0tPe\n8GfgrPKJZRp2NP2S5NIk30ryb0k+NK1QkqTt2XapJ9kF/C3wWuAlwJuTvGhawc680dABehoNHaCn\n0dABGjIaOkBPo6ED9DQaOsBM7eSV+kXAt6vq4ar6KfBPwGXTiTWE0dABehoNHaCn0dABGjIaOkBP\no6ED9DQaOsBM7aTUnwf857qfH+nWSZIGckbeKD333N87E4fZ0k9+8sCgx5ekMyVV23vXO8lvAddU\n1aXdz1cCVVUf3bDfPLytLklPOVU18SlBOyn1s4AHgVcD3wf+FXhzVZ3c1gNKknZs29MvVfV/Sa4A\nbmV1bv4GC12ShrXtV+qSpPkztWu/9PkgUpK/SfLtJMeTHJzWsaeVMckrk/wwyX3d7S/OdMYuxw1J\nlpPcf5p9Bh3LLsNpc87DeCa5IMntSb6R5ESS926x39D/NsfmnJPx3JPkniTHupyHt9hv6PEcm3Me\nxrPLsas7/tEttk82llW14xurTw7fAQ4AzwCOAy/asM/rgC92yy8H7p7Gsaec8ZXA0TOZa4usvw0c\nBO7fYvugYzlBzsHHE9gPHOyWz2H1faC5+rc5Qc7Bx7PLcXb337OAu4GL5m08e+acl/H8c+DvN8uy\nnbGc1iv1Ph9Eugy4EaCq7gEWkuyb0vGnlRHm4AIUVXUX8Ohpdhl6LOmOPS4nDDyeVbVUVce75ceA\nk/zi5ykGH8+eOWE+/n2ufd/bHlbfl9s4hzv4eHbHHpcTBh7PJBcArweu32KXicdyWqXe54NIG/f5\n3ib7zFLfD0u9ovsz54tJfvPMRJvY0GM5ibkZzyQvYPUvi3s2bJqr8TxNTpiD8eymC44BS8BtVXXv\nhl3mYjx75IThx/OvgQ+w9RXVJh5Lr6f+ZF8Hnl9VB1m9rs0/D5znqW5uxjPJOcDngPd1r4Tn0pic\nczGeVbVSVS8FLgBePvST9VZ65Bx0PJP8LrDc/YUWpvRXw7RK/XvA89f9fEG3buM+vzpmn1kam7Gq\nHlv7k62qvgw8I8kvn7mIvQ09lr3My3gm2c1qUf5dVd20yS5zMZ7jcs7LeK7L8z/AHcClGzbNxXiu\n2SrnHIznxcDvJ/l34B+BVyW5ccM+E4/ltEr9XuDXkxxI8kzgj4CN7+QeBd4GP/806g+ranlKx59K\nxvVzVUkuYvWUzx+cwYxPisPWz9xDj+V6W+aco/H8JPDNqvrYFtvnZTxPm3MexjPJc5MsdMt7gUuA\nb23YbfDx7JNz6PGsqqur6vlV9Wus9tHtVfW2DbtNPJZTufZLbfFBpCSXr26uT1TVl5K8Psl3gB8D\n75zGsaeZEfiDJH8K/BT4X+APz2TGNUk+DRwCnpPku8Bh4JnMyVj2zckcjGeSi4G3Aie6+dUCrmb1\nLKi5Gc8+OZmD8QTOB45k9dLbu4DPdOM3N/+v983JfIznL9jpWPrhI0lqiG+USlJDLHVJaoilLkkN\nsdQlqSGWuiQ1xFKXpIZY6pLUEEtdkhry/6KFdj/v/aQiAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x9bb0390>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.hist(sampled_scaling_factors)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**NOW** We will organise our samples based on the number of servers we're running - effectively creating batches"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 0.21828055,  0.36392667,  0.07106982,  0.35537556],\n",
       "       [ 0.49925756,  2.28313128,  2.44344286,  1.35232922],\n",
       "       [ 0.17703713,  0.24183259,  0.00623003,  0.36957271],\n",
       "       [ 1.00505532,  3.98117835,  0.21893658,  0.14258378],\n",
       "       [ 0.41440569,  1.00467281,  0.23425366,  0.49765048],\n",
       "       [ 2.48401778,  0.64162755,  0.03718622,  0.92356851],\n",
       "       [ 1.00677409,  0.57647994,  0.98670485,  0.43695266],\n",
       "       [ 1.70815562,  0.51411446,  3.41275018,  0.42179291],\n",
       "       [ 0.67056163,  0.05074394,  0.39060761,  0.36341694],\n",
       "       [ 0.11971328,  0.58866828,  0.63181065,  2.05749534],\n",
       "       [ 0.08459884,  0.82368285,  0.30062613,  1.906684  ],\n",
       "       [ 0.5379337 ,  0.8772698 ,  1.87724781,  0.81314884],\n",
       "       [ 1.5228271 ,  0.26858821,  0.42374207,  1.94993642],\n",
       "       [ 3.68202781,  1.04852028,  0.84775752,  0.36325295],\n",
       "       [ 0.11497461,  0.0120431 ,  1.58120889,  0.78542681],\n",
       "       [ 0.09562667,  0.42759238,  1.32524812,  1.17473154],\n",
       "       [ 1.62626417,  3.2516898 ,  0.18496208,  1.68430385],\n",
       "       [ 0.07515113,  1.58519883,  0.42267374,  2.33612248],\n",
       "       [ 1.27921425,  0.92020851,  1.16684069,  3.00723003],\n",
       "       [ 1.34674872,  0.42039373,  0.30650172,  1.07809742],\n",
       "       [ 0.20273928,  0.36960488,  1.13041175,  0.49695429],\n",
       "       [ 0.74731115,  1.85200001,  0.06864933,  0.18548627],\n",
       "       [ 0.19659477,  1.11944598,  0.59404985,  0.82596846],\n",
       "       [ 0.2110298 ,  1.72046115,  1.07363669,  2.3335147 ],\n",
       "       [ 0.32605006,  1.53509757,  0.3124965 ,  0.47407891]])"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "samples = sampled_scaling_factors.reshape(NUMBER_OF_SIMULATIONS/len(ports),len(ports))\n",
    "samples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we iterate over our `samples` array now, we'll get groups of four. (The `break` statement stops after the first itertation of the loop)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0.21828055  0.36392667  0.07106982  0.35537556]\n"
     ]
    }
   ],
   "source": [
    "for row in samples:\n",
    "    print(row)\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Importantly**, In switching on the output recording, we need to do so on each of the running servers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Store our time series criteria in a variable to use it in configuring recording and retrieving results\n",
    "ts_match_criteria = {'NetworkElement':'Recreational Lake','RecordingVariable':'Spill Volume'}\n",
    "\n",
    "for v in vs:\n",
    "    v.configure_recording(enable=[ts_match_criteria])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we want to trigger all our runs. We'll use the `run_async=True` option and wait for each group of runs to finish before starting the next group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Completed 4 runs\n",
      "Completed 8 runs\n",
      "Completed 12 runs\n",
      "Completed 16 runs\n",
      "Completed 20 runs\n",
      "Completed 24 runs\n",
      "Completed 28 runs\n",
      "Completed 32 runs\n",
      "Completed 36 runs\n",
      "Completed 40 runs\n",
      "Completed 44 runs\n",
      "Completed 48 runs\n",
      "Completed 52 runs\n",
      "Completed 56 runs\n",
      "Completed 60 runs\n",
      "Completed 64 runs\n",
      "Completed 68 runs\n",
      "Completed 72 runs\n",
      "Completed 76 runs\n",
      "Completed 80 runs\n",
      "Completed 84 runs\n",
      "Completed 88 runs\n",
      "Completed 92 runs\n",
      "Completed 96 runs\n",
      "Completed 100 runs\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>ScalingFactor</th>\n",
       "      <th>SpillVolume</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.218281</td>\n",
       "      <td>9.796761e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.363927</td>\n",
       "      <td>1.740802e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.071070</td>\n",
       "      <td>2.451256e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.355376</td>\n",
       "      <td>1.695799e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.499258</td>\n",
       "      <td>2.456641e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2.283131</td>\n",
       "      <td>1.205902e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2.443443</td>\n",
       "      <td>1.292749e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1.352329</td>\n",
       "      <td>7.028707e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>0.177037</td>\n",
       "      <td>7.674732e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.241833</td>\n",
       "      <td>1.101763e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>0.006230</td>\n",
       "      <td>2.713847e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>0.369573</td>\n",
       "      <td>1.770530e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>1.005055</td>\n",
       "      <td>5.160485e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>3.981178</td>\n",
       "      <td>2.127532e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0.218937</td>\n",
       "      <td>9.830682e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>0.142584</td>\n",
       "      <td>5.925507e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>0.414406</td>\n",
       "      <td>2.007063e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>1.004673</td>\n",
       "      <td>5.158431e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>0.234254</td>\n",
       "      <td>1.062419e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>0.497650</td>\n",
       "      <td>2.448105e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>2.484018</td>\n",
       "      <td>1.314737e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>0.641628</td>\n",
       "      <td>3.214637e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>0.037186</td>\n",
       "      <td>9.903045e+07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>0.923569</td>\n",
       "      <td>4.723210e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>1.006774</td>\n",
       "      <td>5.169713e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>0.576480</td>\n",
       "      <td>2.867369e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>0.986705</td>\n",
       "      <td>5.061978e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>0.436953</td>\n",
       "      <td>2.126279e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>1.708156</td>\n",
       "      <td>8.948622e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>29</th>\n",
       "      <td>0.514114</td>\n",
       "      <td>2.535574e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>70</th>\n",
       "      <td>0.422674</td>\n",
       "      <td>2.050761e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>71</th>\n",
       "      <td>2.336122</td>\n",
       "      <td>1.234604e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>72</th>\n",
       "      <td>1.279214</td>\n",
       "      <td>6.634810e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>0.920209</td>\n",
       "      <td>4.705188e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>74</th>\n",
       "      <td>1.166841</td>\n",
       "      <td>6.029948e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75</th>\n",
       "      <td>3.007230</td>\n",
       "      <td>1.598490e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>76</th>\n",
       "      <td>1.346749</td>\n",
       "      <td>6.998636e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>77</th>\n",
       "      <td>0.420394</td>\n",
       "      <td>2.038710e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>0.306502</td>\n",
       "      <td>1.439051e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>79</th>\n",
       "      <td>1.078097</td>\n",
       "      <td>5.552829e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>80</th>\n",
       "      <td>0.202739</td>\n",
       "      <td>8.994517e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>81</th>\n",
       "      <td>0.369605</td>\n",
       "      <td>1.770700e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>82</th>\n",
       "      <td>1.130412</td>\n",
       "      <td>5.834038e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>0.496954</td>\n",
       "      <td>2.444407e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>84</th>\n",
       "      <td>0.747311</td>\n",
       "      <td>3.779231e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>85</th>\n",
       "      <td>1.852000</td>\n",
       "      <td>9.725927e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>86</th>\n",
       "      <td>0.068649</td>\n",
       "      <td>2.340223e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>87</th>\n",
       "      <td>0.185486</td>\n",
       "      <td>8.107517e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88</th>\n",
       "      <td>0.196595</td>\n",
       "      <td>8.678360e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>89</th>\n",
       "      <td>1.119446</td>\n",
       "      <td>5.775079e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>90</th>\n",
       "      <td>0.594050</td>\n",
       "      <td>2.960949e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>91</th>\n",
       "      <td>0.825968</td>\n",
       "      <td>4.200187e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>92</th>\n",
       "      <td>0.211030</td>\n",
       "      <td>9.422201e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93</th>\n",
       "      <td>1.720461</td>\n",
       "      <td>9.015095e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>94</th>\n",
       "      <td>1.073637</td>\n",
       "      <td>5.528860e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>95</th>\n",
       "      <td>2.333515</td>\n",
       "      <td>1.233192e+10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>96</th>\n",
       "      <td>0.326050</td>\n",
       "      <td>1.541622e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>97</th>\n",
       "      <td>1.535098</td>\n",
       "      <td>8.014328e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>98</th>\n",
       "      <td>0.312496</td>\n",
       "      <td>1.470485e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>99</th>\n",
       "      <td>0.474079</td>\n",
       "      <td>2.322995e+09</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>100 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    ScalingFactor   SpillVolume\n",
       "0        0.218281  9.796761e+08\n",
       "1        0.363927  1.740802e+09\n",
       "2        0.071070  2.451256e+08\n",
       "3        0.355376  1.695799e+09\n",
       "4        0.499258  2.456641e+09\n",
       "5        2.283131  1.205902e+10\n",
       "6        2.443443  1.292749e+10\n",
       "7        1.352329  7.028707e+09\n",
       "8        0.177037  7.674732e+08\n",
       "9        0.241833  1.101763e+09\n",
       "10       0.006230  2.713847e+06\n",
       "11       0.369573  1.770530e+09\n",
       "12       1.005055  5.160485e+09\n",
       "13       3.981178  2.127532e+10\n",
       "14       0.218937  9.830682e+08\n",
       "15       0.142584  5.925507e+08\n",
       "16       0.414406  2.007063e+09\n",
       "17       1.004673  5.158431e+09\n",
       "18       0.234254  1.062419e+09\n",
       "19       0.497650  2.448105e+09\n",
       "20       2.484018  1.314737e+10\n",
       "21       0.641628  3.214637e+09\n",
       "22       0.037186  9.903045e+07\n",
       "23       0.923569  4.723210e+09\n",
       "24       1.006774  5.169713e+09\n",
       "25       0.576480  2.867369e+09\n",
       "26       0.986705  5.061978e+09\n",
       "27       0.436953  2.126279e+09\n",
       "28       1.708156  8.948622e+09\n",
       "29       0.514114  2.535574e+09\n",
       "..            ...           ...\n",
       "70       0.422674  2.050761e+09\n",
       "71       2.336122  1.234604e+10\n",
       "72       1.279214  6.634810e+09\n",
       "73       0.920209  4.705188e+09\n",
       "74       1.166841  6.029948e+09\n",
       "75       3.007230  1.598490e+10\n",
       "76       1.346749  6.998636e+09\n",
       "77       0.420394  2.038710e+09\n",
       "78       0.306502  1.439051e+09\n",
       "79       1.078097  5.552829e+09\n",
       "80       0.202739  8.994517e+08\n",
       "81       0.369605  1.770700e+09\n",
       "82       1.130412  5.834038e+09\n",
       "83       0.496954  2.444407e+09\n",
       "84       0.747311  3.779231e+09\n",
       "85       1.852000  9.725927e+09\n",
       "86       0.068649  2.340223e+08\n",
       "87       0.185486  8.107517e+08\n",
       "88       0.196595  8.678360e+08\n",
       "89       1.119446  5.775079e+09\n",
       "90       0.594050  2.960949e+09\n",
       "91       0.825968  4.200187e+09\n",
       "92       0.211030  9.422201e+08\n",
       "93       1.720461  9.015095e+09\n",
       "94       1.073637  5.528860e+09\n",
       "95       2.333515  1.233192e+10\n",
       "96       0.326050  1.541622e+09\n",
       "97       1.535098  8.014328e+09\n",
       "98       0.312496  1.470485e+09\n",
       "99       0.474079  2.322995e+09\n",
       "\n",
       "[100 rows x 2 columns]"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "spill_results=[]\n",
    "total_runs=0\n",
    "\n",
    "for group in samples:\n",
    "    group_run_responses = [] # Somewhere\n",
    "    for i in range(len(vs)): # Will be 0,1.. #ports\n",
    "        total_runs += 1\n",
    "        scaling_factor = group[i]\n",
    "        v = vs[i]\n",
    "\n",
    "        # We are running the multiple many times in this case - so lets drop any results we already have...\n",
    "        v.drop_all_runs()\n",
    "\n",
    "        # Set $InflowScaling to current scaling factor\n",
    "        v.update_function('$InflowScaling',scaling_factor)\n",
    "        response = v.run_model(run_async=True)\n",
    "        group_run_responses.append(response)\n",
    "\n",
    "    #### NOW, All runs for this group have been triggered. Now go back and retrieve results\n",
    "    \n",
    "    # Retrieve the spill time series, as an annual sum, with the column named for the variable ('Spill Volume')\n",
    "    for i in range(len(vs)): # Will be 0,1.. #ports\n",
    "        scaling_factor = group[i]\n",
    "        v = vs[i]\n",
    "        r = group_run_responses[i]\n",
    "        \n",
    "        code = r.getresponse().getcode() # Wait until the job is finished\n",
    "\n",
    "        run_results = v.retrieve_multiple_time_series(criteria=ts_match_criteria,timestep='annual',name_fn=veneer.name_for_variable)\n",
    "    \n",
    "        # Store the mean spill volume and the scaling factor we used\n",
    "        spill_results.append({'ScalingFactor':scaling_factor,'SpillVolume':run_results['Spill Volume'].mean()})\n",
    "    veneer.log('Completed %d runs'%total_runs)\n",
    "    \n",
    "# Convert the results to a Data Frame\n",
    "import pandas as pd\n",
    "spill_results_df = pd.DataFrame(spill_results)\n",
    "spill_results_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x9d71dd8>"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAEACAYAAABfxaZOAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAFitJREFUeJzt3X+sZHV9xvHnWdYuILK7atnbuLIjNQE0kgtV0gQbx6qw\naiuEpBatxbVqiCkKpbEsJGZbowZMXDFt+UeodyE1ak35lRhZyHI0mPJD2SsLIqWpi8W6F3UBS01R\nuJ/+cc7wnb3c3Tt35/y6c96vZHLne+6ZOd/57OznnnnOnBlHhAAA3bCq6QkAAOpD0weADqHpA0CH\n0PQBoENo+gDQITR9AOiQkZu+7VW2d9u+uRhvs/2Y7fuKy+bqpgkAKMPqZax7kaQHJR07tGx7RGwv\nd0oAgKqMtKdve6Okd0i6ZuGvSp8RAKAyo8Y7n5f0cUkLT9+90Pas7Wtsry13agCAsi3Z9G2/U9Jc\nRMzqwD37qyWdEBHTkvZJIuYBgJbzUp+9Y/szkt4n6VlJR0l6iaR/jYjzh9bZJOmWiDhlkdvz4T4A\ncBgiovQIfck9/Yi4PCKOj4gTJJ0naVdEnG97ami1cyU9cIj74BKhbdu2NT6HtlyoBbWgFoe+VGU5\n795Z6LO2pyXNS9or6YJSZjTB9u7d2/QUWoNaJNQioRbVW1bTj4hvSfpWcf38JVYHALQMZ+TWaMuW\nLU1PoTWoRUItEmpRvSUP5I69ATuq3gYATBrbiiYO5KI8WZY1PYXWoBYJtUioRfVo+gDQIcQ7ANBC\nxDsAgLHR9GtEXplQi4RaJNSiejR9AOgQMn0AaCEyfQDA2Gj6NSKvTKhFQi0SalE9mj4AdAiZPgC0\nEJk+AGBs43ye/sieeuqpOjYjSVqzZo2OPPLI2ra3HFmWqd/vNz2NVqAWCbVIqEX1amn6xx13fB2b\nkRRau3adHn/8xzVtDwBWlpEzfdurJH1X0mMR8S7b6yV9VdIm5d+c9e6IeMEuff4duXVl+r/WEUcc\no2ef/XVN2wOAarQh079I0g+Gxlsl3R4RJ0raJemyMicGACjfSE3f9kZJ75B0zdDisyXtKK7vkHRO\nuVObPLwHOaEWCbVIqEX1Rt3T/7ykj+vAnGZDRMxJUkTsk3RcyXMDAJRsyQO5tt8paS4iZm33D7Hq\nIYL7LZJ6xfV1kqYlDe4qK36WM46YP+AdAIM9hzaM+/1+q+bDuD3jgbbMp6nxYFlb5lPnOMsyzczM\nSJJ6vZ6qsuSBXNufkfQ+Sc9KOkrSSyTdIOn1kvoRMWd7StIdEXHyIrfnQC4ALFNjB3Ij4vKIOD4i\nTpB0nqRdEfHnkm5RvgsvSe+XdFPZk5s0C/fquoxaJNQioRbVG+eM3Cskvc32w5LeUowBAC1Wy2fv\nEO8AwPK04X36AIAVjqZfI/LKhFok1CKhFtWj6QNAh5DpA0ALkekDAMZG068ReWVCLRJqkVCL6tH0\nAaBDyPQBoIXI9AEAY6Pp14i8MqEWCbVIqEX1aPoA0CFk+gDQQmT6AICx0fRrRF6ZUIuEWiTUono0\nfQDoEDJ9AGihxjJ922ts3217t+09trcVy7fZfsz2fcVlc9mTAwCUa5TvyH1G0psj4lRJ05Lebvv0\n4tfbI+K04vLNKic6CcgrE2qRUIuEWlRvpEw/In5VXF0jabVSXlP6Sw8AQHVGyvRtr5L0PUm/K+kf\nI+KyIubZIukpSd+V9NcR8dQityXTB4BlqirTXz3KShExL+lU28dKusH2ayRdLemTERG2PyVpu6QP\nLn4PWyT1iuvrlKdE/WKcFT/LGUfMK8sy9fv5ePBykTFjxozbPM6yTDMzM5KkXq+nqiz73Tu2PyHp\nfyNi+9CyTZJuiYhTFlmfPf1CNvTHqOuoRUItEmqRNPnunZfbXltcP0rS2yT90PbU0GrnSnqg7MkB\nAMq15J6+7ddJ2qH8D8QqSV+NiE/bvk55TjMvaa+kCyJibpHbs6cPAMtU1Z4+J2cBQAvxgWsTYHDQ\nBtRiGLVIqEX1aPoA0CHEOwDQQsQ7AICx0fRrRF6ZUIuEWiTUono0fQDoEDJ9AGghMn0AwNho+jUi\nr0yoRUItEmpRPZo+AHQImT4AtBCZPgBgbDT9GpFXJtQioRYJtageTR8AOoRMHwBaiEwfADC2Ub4u\ncY3tu23vtr3H9rZi+XrbO20/bPvWwVcq4uDIKxNqkVCLhFpUb8mmHxHPSHpzRJyq/OsR3277dElb\nJd0eESdK2iXpskpnCgAY27IyfdtHS/q2pI9Iul7SmyJirviS9CwiTlrkNmT6ALBMjWb6tlfZ3i1p\nn6TbIuJeSRsGX4QeEfskHVf25AAA5Vo9ykoRMS/pVNvHSrrB9mv1wt33Q+zOb5HUK66vU54S9Ytx\nVvwsZxwxryzL1O/n40FG2IbxcF7Zhvk0OR4sa8t8mhzPzs7q4osvbs18mhxfddVVmp6ebs186hxn\nWaaZmRlJUq/XU1WW/ZZN25+Q9CtJH5LUH4p37oiIkxdZn3inkA39Meo6apFQi4RaJFXFO0s2fdsv\nl/SbiHjK9lGSbpV0haQ3SdofEVfavlTS+ojYusjtafoAsExVNf1R4p3fkbTD9irlxwC+GhHfsH2X\npK/Z/gtJj0p6d9mTAwCUa5S3bO6JiNMiYjoiTomITxfL90fEWyPixIg4MyKerH66K9twnt111CKh\nFgm1qB5n5AJAh/DZOwDQQnz2DgBgbDT9GpFXJtQioRYJtageTR8AOoRMHwBaiEwfADA2mn6NyCsT\napFQi4RaVI+mDwAdQqYPAC1Epg8AGBtNv0bklQm1SKhFQi2qR9MHgA4h0weAFiLTBwCMbcmmb3uj\n7V22H7S9x/ZHi+XbbD9m+77isrn66a5s5JUJtUioRUItqjfKN2c9K+mSiJi1fYyk79m+rfjd9ojY\nXt30AABlOpwvRr9R0t9LeqOkpyPic0usT6YPAMvUikzfdk/StKS7i0UX2p61fY3ttSXPDQBQspGb\nfhHtfF3SRRHxtKSrJZ0QEdOS9kki5lkCeWVCLRJqkVCL6o2S6cv2auUN//qIuEmSIuJnQ6t8UdIt\nB7+HLZJ6xfV1yl8s9ItxVvwsZxwxryzL1O/n48GTiHG7xgNtmU+T49nZ2VbNp8nx7Oxsq+ZT5zjL\nMs3MzEiSer2eqjJSpm/7Okk/j4hLhpZNRcS+4vpfSXpDRLx3kduS6QPAMlWV6S/Z9G2fIenbkvYo\n794h6XJJ71W+yz4vaa+kCyJibpHb0/QBYJkaO5AbEd+JiCMiYjoiTo2I0yLimxFxfkScUiw/Z7GG\njwMtjDa6jFok1CKhFtXjjFwA6BA+ewcAWqgV79MHAKxsNP0akVcm1CKhFgm1qB5NHwA6hEwfAFqI\nTB8AMDaafo3IKxNqkVCLhFpUj6YPAB1Cpg8ALUSmDwAYG02/RuSVCbVIqEVCLapH0weADiHTB4AW\nItMHAIyNpl8j8sqEWiTUIqEW1Vuy6dveaHuX7Qdt77H9sWL5ets7bT9s+1bba6ufLgBgHKN8XeKU\npKmImLV9jKTvSTpb0gck/SIiPmv7UknrI2LrIrcn0weAZWry6xL3RcRscf1pSQ9J2qi88e8oVtsh\n6ZyyJwcAKNeyMn3bPeVfhn6XpA2D78WNiH2Sjit7cpOGvDKhFgm1SKhF9UZu+kW083VJFxV7/Asz\nm7oyHADAYVo9ykq2Vytv+NdHxE3F4jnbGyJirsj9Hz/4PWyR1Cuur1P+YqFfjLPiZznjiHllWaZ+\nPx8P9hzaMO73+62aD+P2jAfaMp+mxoNlbZlPneMsyzQzMyNJ6vV6qspIJ2fZvk7SzyPikqFlV0ra\nHxFXciAXAMrV2IFc22dI+jNJf2h7t+37bG+WdKWkt9l+WNJbJF1R9uQmzcK9ui6jFgm1SKhF9ZaM\ndyLiO5KOOMiv31rudAAAVeKzdwCghfjsHQDA2Gj6NSKvTKhFQi0SalE9mj4AdAiZPgC0EJk+AGBs\nNP0akVcm1CKhFgm1qB5NHwA6hEwfAFqITB8AMLaJa/rPPXeEbNd6mZrqjTQ38sqEWiTUIqEW1Rvp\no5VXlv9T3R/tPzdX+iswAKjExGX60hrV/30uVtV1BNAtZPoAgLHR9GtEXplQi4RaJNSiejR9AOiQ\nJTN929dK+iNJcxFxSrFsm6QPK30v7uUR8c2D3J5MHwCWqclM/0uSzlpk+faIOK24LNrwAQDtsmTT\nj4g7JT2xyK94n+IykVcm1CKhFgm1qN44mf6FtmdtX2N7bWkzAgBU5nBPzrpa0icjImx/StJ2SR88\n+OpbJPWK6+skTUvqF+Os+FnWeLCsqvtfbPwi2fW98Fm/foP279+Xb73YM+r3+4xX8HigLfNpajxY\n1pb51DnOskwzMzOSpF6vp6qMdHKW7U2SbhkcyB31d8XvO3Egt95tcuAYmHRNn5xlDWX4tqeGfneu\npAfKnNTkypqeQGuQ3SbUIqEW1Vsy3rH9ZeVZxsts/1jSNklvtj0taV7SXkkXVDhHAEBJ+OydUhDv\nAChX0/EOAGAC0PRrlTU9gdYgu02oRUItqkfTB4AOIdMvBZk+gHKR6QMAxkbTr1XW9ARag+w2oRYJ\ntageTR8AOoRMvxRk+gDKRaYPABgbTb9WWdMTaA2y24RaJNSiejR9AOgQMv1SkOkDKBeZPgBgbDT9\nWmVNT6A1yG4TapFQi+rR9AGgQ8j0S0GmD6BcjWX6tq+1PWf7/qFl623vtP2w7Vttry17YgCA8o0S\n73xJ0lkLlm2VdHtEnChpl6TLyp7YZMqankBrkN0m1CKhFtVbsulHxJ2Snliw+GxJO4rrOySdU/K8\nAAAVGCnTt71J0i0RcUox3h8RLx36/QHjBbcl069ge2T6wGSrKtNfXdL9LNGBtkjqFdfXSZqW1C/G\nWfGzrPFgWVX3f7Cxlvh9NdsbvBzu9/uMGTNeweMsyzQzMyNJ6vV6qsrh7uk/JKkfEXO2pyTdEREn\nH+S27Ok/L9OBf5wOf3srfU8/y7Lnn/hdRy0SapE0fUaui8vAzcp33yXp/ZJuKnFOAICKLLmnb/vL\nyndPXyZpTtI2STdK+hdJr5T0qKR3R8STB7k9e/oVbG+l7+kDOLSq9vQ5OasUNH0A5Wo63kEpspLu\nZ41s13qZmuqVNPfc4AAWqMUwalG9st69g1o9o7pfzczNlb7DAaABxDulqD/eaeIxEikB9SHeAQCM\njaZfq6zpCbQG2W1CLRJqUT2aPgB0CJl+Kcj0AZSLTB8AMDaafq2ypifQGmS3CbVIqEX1aPoA0CFk\n+qXoQqZ/pPKTwuqxYcMm7du3t7btAW3DZ++MhKY/OdvkwDG6jQO5EyFregItkjU9gdYgx06oRfVo\n+gDQIcQ7pSDeqWJ7xDvoMuIdAMDYxmr6tvfa/r7t3bbvKWtSkytregItkjU9gdYgx06oRfXG/Tz9\neeVfkP5EGZMBAFRrrEzf9o8kvT4ifnGIdcj0V/z2mtgmmT66ra2Zfki6zfa9tj9cxoQAANUZN945\nIyJ+avu3lTf/hyLizheutkVSr7i+TtK0pH4xzoqfZY0Hy6q6/4ONtcTv+wvWrWN7ZY7L3t5g2aG3\nN8h4+/3+xI5nZ2d18cUXt2Y+TY6vuuoqTU9Pt2Y+dY6zLNPMzIwkqdfrqSqlvWXT9jZJ/xMR2xcs\nJ955XqYD/zhVvb0ylb3NTIeuRXfinSzLnm8CXUctktZ9DIPtoyWtioinbb9Y0k5JfxcROxesR9Nf\n8dtrYpvdafrAYqpq+uPEOxsk3ZA3da2W9M8LGz4AoF0O+0BuRPwoIqYj4tSIeF1EXFHmxCZT1vQE\nWiRregKtwXvTE2pRvXEP5AIVWSO79Fe2h8THOaML+OydUpDpr/zt5dvkOALaoq3v0wcArCA0/Vpl\nTU+gRbKmJ9Aa5NgJtageTR8AOoRMvxTdyLu78BjJ9NEWZPoAgLHR9GuVNT2BFsmankBrkGMn1KJ6\nNH0A6BAy/VJ0I+/uwmMk00dbtPGzd4AJU+9ZwE2cATw11dPc3KO1bW/VqqM1P/+r2rYncWb1Uoh3\napU1PYEWyZqewCKeUf7qop7LoPnWmWPn26zvMeYNfzm3uWPsbdb5R20loukDQIeQ6ZeiG3k3j7H8\n7dV9DCGPryb/33ESjs3wPn0AwNjGavq2N9v+oe1/t31pWZOaXFnTE2iRrOkJtEB+4LjOS/tlTU9g\n4h1207e9StI/SDpL0mslvcf2SWVNbDLNNj2BFqEW6cDx51XfwdW243lRtXH29E+X9EhEPBoRv5H0\nFUlnlzOtSfVk0xNoEWqRUIuEWlRtnKb/Ckn/NTR+rFgGAGipWk7OOvbYP65jM5Lm9ctf1rSpw7K3\n6Qm0yN6mJ9Aie5ueQIvsbXoCE++w37Jp+/cl/W1EbC7GWyVFRFy5YL2VECQCQOtU8ZbNcZr+EZIe\nlvQWST+VdI+k90TEQ+VNDwBQpsOOdyLiOdsXStqp/NjAtTR8AGi3ys/IBQC0R2Vn5HblxC3be21/\n3/Zu2/cUy9bb3mn7Ydu32l47tP5lth+x/ZDtM4eWn2b7/qJeVzXxWJbL9rW252zfP7SstMdu+7ds\nf6W4zb/ZPr6+R7c8B6nFNtuP2b6vuGwe+t0k12Kj7V22H7S9x/bHiuWde24sUouPFsube25EROkX\n5X9M/kPSJkkvUn7GxUlVbKvpi6T/lLR+wbIrJf1Ncf1SSVcU118jabfyWK1X1GjwautuSW8orn9D\n0llNP7YRHvsbJU1Lur+Kxy7pI5KuLq7/qaSvNP2Yl1mLbZIuWWTdkye8FlOSpovrxyg/9ndSF58b\nh6hFY8+Nqvb0u3TilvXCV0xnS9pRXN8h6Zzi+ruU/4M8GxF7JT0i6XTbU5JeEhH3FutdN3Sb1oqI\nOyU9sWBxmY99+L6+rvxNA610kFpI+fNjobM12bXYFxGzxfWnJT0kaaM6+Nw4SC0G5zM18tyoqul3\n6cStkHSb7Xttf6hYtiEi5qT8H13SccXyhXX5SbHsFcprNLCS63VciY/9+dtExHOSnrT90uqmXokL\nbc/avmYozuhMLWz3lL8Cukvl/r9YcfUYqsXdxaJGnht8yub4zoiI0yS9Q9Jf2v4DvfBDTrp8tLzM\nx74SPjFs2NWSToiIaUn7JH2uxPtufS1sH6N8z/OiYi+3yv8Xra7HIrVo7LlRVdP/iaThgwkbi2UT\nJyJ+Wvz8maQblUdbc7Y3SFLxsuzxYvWfSHrl0M0HdTnY8pWozMf+/O+cnxdybETsr27q5YqIn0UR\ntEr6ovLnhtSBWtherbzJXR8RNxWLO/ncWKwWTT43qmr690p6te1Ntn9L0nmSbq5oW42xfXTxF1y2\nXyzpTEl7lD/WLcVq75c0eNLfLOm84mj7qyS9WtI9xUvdp2yfbtuSzh+6TdtZB+5ZlPnYby7uQ5L+\nRNKuyh5FOQ6oRdHYBs6V9EBxvQu1+CdJP4iILwwt6+pz4wW1aPS5UeFR683Kj1Q/ImlrE0fOq75I\nepXydybtVt7stxbLXyrp9uLx75S0bug2lyk/Iv+QpDOHlv9ecR+PSPpC049txMf/ZUn/rfwzgn8s\n6QOS1pf12JV/DdrXiuV3Seo1/ZiXWYvrJN1fPEduVJ5pd6EWZ0h6buj/xn1FPyjt/8VKqcchatHY\nc4OTswCgQziQCwAdQtMHgA6h6QNAh9D0AaBDaPoA0CE0fQDoEJo+AHQITR8AOuT/AUuFGRIcH2sA\nAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x9d03748>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "spill_results_df['SpillVolumeGL'] = spill_results_df['SpillVolume'] * 1e-6 # Convert to GL\n",
    "spill_results_df['SpillVolumeGL'].hist()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Final remarks\n",
    "\n",
    "The above example isn't as efficient as it could be, but it may be good enough for many circumstances.\n",
    "\n",
    "The simulations run in parallel, but everything else (configuring recorders, retrieving and post-processing results) is done sequentially. (And no simulations are taking place while that's happening). \n",
    "\n",
    "Furthermore, if some model runs complete quicker than others, one or more Veneer servers will be idle waiting for further instructions.\n",
    "\n",
    "The example above is a reasonable approach if the simulations take much longer than the post processing and if the simulations will typically take around the same amount of time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Terminate the veneer servers\n",
    "kill_all_now(processes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}