Meetup11-Lab4-BatchAI-MultiNode-HyperParam-Search.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LAB 4 - Batch AI  - Random Search CNTK GPU (Multi-GPU, Multi-node)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "This example shows how to perform random search hyperparameter tuning using CNTK with MNIST dataset to train a convolutional neural network (CNN) on a GPU cluster.\n",
    "\n",
    "## Details\n",
    "\n",
    "- We provide a CNTK example ConvMNIST.py to accept command line arguments for CNTK dataset, model locations, model file suffix and two hyperparameters for tuning: 1. hidden layer dimension and 2. feedforward constant\n",
    "- For demonstration purposes, MNIST dataset and CNTK training script will be deployed at Azure File Share;\n",
    "- Standard output of the job and the model will be stored on Azure File Share;\n",
    "- MNIST dataset (http://yann.lecun.com/exdb/mnist/) has been preprocessed by usign install_mnist.py available here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instructions\n",
    "\n",
    "### Create in Azure the Resource Groups and Storage Accounts needed.\n",
    "```\n",
    "> ssh sshuser@YOUR.VM.IP.ADDRESS\n",
    "> sudo pip install azure\n",
    "> sudo pip3 install azure-mgmt-batchai --upgrade\n",
    "> az login\n",
    "> az group create --name batchai_rg  --location eastus\n",
    "> az storage account create --location eastus --name batchaipablo --resource-group batchai_rg --sku Standard_LRS\n",
    "> az storage account keys list --account-name batchaipablo --resource-group batchai_rg -o table\n",
    "> az ad sp create-for-rbac --name MyAppSvcPppl --password Passw0rd\n",
    "> az storage account keys list --account-name batchaipablo --resource-group batchai_rg\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read Configuration and Create Batch AI client"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "nbpresent": {
     "id": "bfa11f00-8866-4051-bbfe-a9646e004910"
    }
   },
   "outputs": [],
   "source": [
    "from __future__ import print_function\n",
    "\n",
    "from datetime import datetime\n",
    "import os\n",
    "import sys\n",
    "import zipfile\n",
    "import numpy\n",
    "import queue\n",
    "import threading\n",
    "import requests\n",
    "\n",
    "from azure.storage.file import FileService\n",
    "from azure.storage.blob import BlockBlobService\n",
    "import azure.mgmt.batchai.models as models\n",
    "\n",
    "# utilities.py contains helper functions\n",
    "import utilities\n",
    "import hyperparam_utilities\n",
    "from hyperparam_utilities import Hyperparameter, MetricExtractor, run_then_return_metric\n",
    "\n",
    "# Resource Group\n",
    "location = 'eastus'\n",
    "resource_group = 'batchai_rg'\n",
    "\n",
    "# credentials used for authentication\n",
    "client_id = 'ec0640c7-61fa-4662-bce4-8a3e931939ac'\n",
    "secret = 'Passw0rd'\n",
    "token_uri = 'https://login.microsoftonline.com/72f988bf-86f1-41af-91ab-2d7cd011db47/oauth2/token'\n",
    "subscription_id = 'b1395605-1fe9-4af4-b3ff-82a4725a3791'\n",
    "\n",
    "# credentials used for storage\n",
    "storage_account_name = 'batchaipablo'\n",
    "storage_account_key = 'y59heteYEbw5nTLBB/b7rj3jUphvs2Iwslg4AsXFSb4G7ZLgJUep4AuccSmST7I3E8Zw4BaUloebK+VyKmGpog=='\n",
    "\n",
    "# specify the credentials used to remote login your GPU node\n",
    "admin_user_name = 'sshuser'\n",
    "admin_user_password = 'Passw0rd.1!!'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Keyring cache token has failed: No recommended backend was available. Install the keyrings.alt package if you want to use the non-recommended backends. See README.rst for details.\n"
     ]
    }
   ],
   "source": [
    "from azure.common.credentials import ServicePrincipalCredentials\n",
    "import azure.mgmt.batchai as batchai\n",
    "import azure.mgmt.batchai.models as models\n",
    "\n",
    "creds = ServicePrincipalCredentials(client_id=client_id, secret=secret, token_uri=token_uri)\n",
    "\n",
    "client = batchai.BatchAIManagementClient(credentials=creds,subscription_id=subscription_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Azure Blob Container"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will create a new Blob Container with name 'batchailab4' under your storage account. This will be used to store the input training dataset\n",
    "\n",
    "**Note** You don't need to create new file share for every cluster. We are doing this in this sample to simplify resource management for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "azure_blob_container_name = 'batchailab4'\n",
    "blob_service = BlockBlobService(storage_account_name, storage_account_key)\n",
    "blob_service.create_container(azure_blob_container_name, fail_on_exist=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload MNIST Dataset to Azure Blob Container\n",
    "\n",
    "For demonstration purposes, we will download preprocessed MNIST dataset to the current directory and upload it to Azure Blob Container directory named mnist_dataset.\n",
    "\n",
    "There are multiple ways to create folders and upload files into Azure Blob Container - you can use Azure Portal, Storage Explorer, Azure CLI2 or Azure SDK for your preferable programming language. In this example we will use Azure SDK for python to copy files into Blob."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading MNIST dataset...\n",
      "Done\n"
     ]
    }
   ],
   "source": [
    "mnist_dataset_directory = 'mnist_dataset'\n",
    "utilities.download_and_upload_mnist_dataset_to_blob(\n",
    "    blob_service, azure_blob_container_name, mnist_dataset_directory)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create File Share\n",
    "\n",
    "For this example we will create a new File Share with name `batchailab4` under your storage account. This will be used to share the training script file and output file.\n",
    "\n",
    "\n",
    "**Note** You don't need to create new file share for every cluster. We are doing this in this sample to simplify resource management for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "False"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "azure_file_share_name = 'batchailab4'\n",
    "file_service = FileService(storage_account_name, storage_account_key)\n",
    "file_service.create_share(azure_file_share_name, fail_on_exist=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Upload the training script ConvMNIST.py to file share directory named hyperparam_samples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "cntk_script_path = \"hyperparam_samples\"\n",
    "file_service.create_directory(\n",
    "    azure_file_share_name, cntk_script_path, fail_on_exist=False)\n",
    "file_service.create_file_from_path(\n",
    "    azure_file_share_name, cntk_script_path, 'ConvMNIST.py', 'ConvMNIST.py')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Configure Compute Cluster\n",
    "\n",
    "- For this example we will use a GPU cluster of STANDARD_NC6 nodes. Number of nodes in the cluster is configured with nodes_count variable;\n",
    "- We will mount blob container at folder with name external_ABFS. Full path of this folder on a computer node will be AZ_BATCHAI_MOUNT_ROOT/external_ABFS;\n",
    "- We will mount file share at folder with name external_AFS. Full path of this folder on a computer node will be AZ_BATCHAI_MOUNT_ROOT/external_AFS;\n",
    "- We will call the cluster nc6;\n",
    "\n",
    "So, the cluster will have the following parameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "azure_file_share = 'external_AFS'\n",
    "azure_blob = 'external_ABFS'\n",
    "nodes_count = 4\n",
    "cluster_name = 'nc6'\n",
    "vmsize = \"Standard_NC6\"\n",
    "\n",
    "volumes = models.MountVolumes(\n",
    "    azure_file_shares=[\n",
    "        models.AzureFileShareReference(\n",
    "            account_name=storage_account_name,\n",
    "            credentials=models.AzureStorageCredentialsInfo(\n",
    "                account_key=storage_account_key),\n",
    "            azure_file_url='https://{0}.file.core.windows.net/{1}'.format(\n",
    "                storage_account_name, azure_file_share_name),\n",
    "            relative_mount_path=azure_file_share)\n",
    "    ],\n",
    "    azure_blob_file_systems=[\n",
    "        models.AzureBlobFileSystemReference(\n",
    "            account_name=storage_account_name,\n",
    "            credentials=models.AzureStorageCredentialsInfo(\n",
    "                account_key=storage_account_key),\n",
    "            container_name=azure_blob_container_name,\n",
    "            relative_mount_path=azure_blob)\n",
    "    ]\n",
    ")\n",
    "\n",
    "parameters = models.ClusterCreateParameters(\n",
    "    location=location,\n",
    "    vm_size=vmsize,\n",
    "    virtual_machine_configuration=models.VirtualMachineConfiguration(\n",
    "        image_reference=models.ImageReference(\n",
    "            publisher=\"microsoft-ads\",\n",
    "            offer=\"linux-data-science-vm-ubuntu\",\n",
    "            sku=\"linuxdsvmubuntu\",\n",
    "            version=\"latest\")),    \n",
    "    user_account_settings=models.UserAccountSettings(\n",
    "        admin_user_name=admin_user_name,\n",
    "        admin_user_password=admin_user_password),\n",
    "    scale_settings=models.ScaleSettings(\n",
    "        manual=models.ManualScaleSettings(target_node_count=nodes_count)\n",
    "    ),\n",
    "    node_setup=models.NodeSetup(\n",
    "        mount_volumes=volumes,\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Compute Cluster"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster = client.clusters.create(resource_group, cluster_name, parameters).result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Monitor Cluster Creation\n",
    "\n",
    "Monitor the just created cluster. utilities.py contains a helper function to print out detail status of the cluster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Token expired or is invalid. Attempting to refresh.\n",
      "Keyring cache token has failed: No recommended backend was available. Install the keyrings.alt package if you want to use the non-recommended backends. See README.rst for details.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cluster state: steady Target: 4; Allocated: 4; Idle: 4; Unusable: 0; Running: 0; Preparing: 0; Leaving: 0\n"
     ]
    }
   ],
   "source": [
    "cluster = client.clusters.get(resource_group, cluster_name)\n",
    "utilities.print_cluster_status(cluster)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Parametric Sweeping using Random Search\n",
    "Define the space of hyperparameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "import hyperparam_utilities\n",
    "from hyperparam_utilities import Hyperparameter, MetricExtractor, run_then_return_metric\n",
    "\n",
    "space = {Hyperparameter('feedforward constant', 'feedforward_const', 'log', [0.0001, 10]),\n",
    "         Hyperparameter('hidden layers dimenson', 'hidden_layers_dim', 'choice', [100, 200, 300])}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define the total number of hyperparameter configurations we want to try"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "num_configs = 16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Generate num_configs random hyper-parameter configuration and corresponding index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 : {'hidden_layers_dim': 300, 'feedforward_const': 0.043951574155499704}\n",
      "1 : {'hidden_layers_dim': 300, 'feedforward_const': 0.00013025752255328113}\n",
      "2 : {'hidden_layers_dim': 200, 'feedforward_const': 5.414203512473023}\n",
      "3 : {'hidden_layers_dim': 300, 'feedforward_const': 5.582105156163857}\n",
      "4 : {'hidden_layers_dim': 200, 'feedforward_const': 0.003246553415665837}\n",
      "5 : {'hidden_layers_dim': 200, 'feedforward_const': 0.0019433278974297264}\n",
      "6 : {'hidden_layers_dim': 300, 'feedforward_const': 0.006299208962397572}\n",
      "7 : {'hidden_layers_dim': 100, 'feedforward_const': 0.00037441504751268363}\n",
      "8 : {'hidden_layers_dim': 300, 'feedforward_const': 0.9893207844029632}\n",
      "9 : {'hidden_layers_dim': 100, 'feedforward_const': 0.7318551250004928}\n",
      "10 : {'hidden_layers_dim': 100, 'feedforward_const': 0.716883342271889}\n",
      "11 : {'hidden_layers_dim': 100, 'feedforward_const': 0.0025973455297094625}\n",
      "12 : {'hidden_layers_dim': 100, 'feedforward_const': 0.05601180437954487}\n",
      "13 : {'hidden_layers_dim': 200, 'feedforward_const': 0.00018638512254130432}\n",
      "14 : {'hidden_layers_dim': 300, 'feedforward_const': 0.05848771920419732}\n",
      "15 : {'hidden_layers_dim': 100, 'feedforward_const': 8.214786646778737}\n"
     ]
    }
   ],
   "source": [
    "job_configs = {}\n",
    "for i in range(num_configs):\n",
    "    job_configs[i] = Hyperparameter.get_random_hyperparameter_configuration(space)\n",
    "    print(str(i) + ' : ' + str(job_configs[i]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following helper function is used to construct the job creation parameters with given hyperparameter configuration"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "def generate_job_create_parameters(configs):\n",
    "    environment_variables=[]\n",
    "    for config in configs:\n",
    "        environment_variables.append(models.EnvironmentVariable(\n",
    "                name='HYPERPARAM_'+config,\n",
    "                value=str(configs[config])))\n",
    "\n",
    "    parameter =models.JobCreateParameters(\n",
    "        location=location,\n",
    "        cluster=models.ResourceId(id=cluster.id),\n",
    "        node_count=1,\n",
    "        std_out_err_path_prefix='$AZ_BATCHAI_MOUNT_ROOT/{0}'.format(azure_file_share),\n",
    "        environment_variables=environment_variables,\n",
    "        output_directories=[\n",
    "            models.OutputDirectory(\n",
    "                id='ALL',\n",
    "                path_prefix='$AZ_BATCHAI_MOUNT_ROOT/{0}'.format(azure_file_share))\n",
    "        ],\n",
    "        cntk_settings=models.CNTKsettings(\n",
    "            python_script_file_path='$AZ_BATCHAI_MOUNT_ROOT/{0}/{1}/ConvMNIST.py'.format(azure_file_share, cntk_script_path),\n",
    "            command_line_args='--datadir {0} --outputdir $AZ_BATCHAI_OUTPUT_ALL --logdir $AZ_BATCHAI_OUTPUT_ALL --epochs 16 --feedforward_const $HYPERPARAM_feedforward_const --hidden_layers_dim $HYPERPARAM_hidden_layers_dim'.format(\n",
    "                '$AZ_BATCHAI_MOUNT_ROOT/{0}/{1}'.format(azure_blob, mnist_dataset_directory))\n",
    "        )\n",
    "    )\n",
    "    return parameter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We define the following metric extractor to extract desired metric from learning log file.\n",
    "- In this example, we extract the number between \"metric =\" and \"%\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "metric_extractor = MetricExtractor(\n",
    "                        list_option='ALL',\n",
    "                        logfile='progress.log',\n",
    "                        regex='metric =(.*?)\\%')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each configuration, we generate specific job creation parameters with given configuration and number of epochs.\n",
    "\n",
    "A new thread is started per new job that submits and monitors the job. Once job completes, the final metric is extracted and returned from log file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Submitting 16 jobs with 16 configurations \n",
      "Job 46d4cd95 has completed for config 7\n",
      "Job 0b075c76 has completed for config 14\n",
      "Job 3781764d has completed for config 9\n",
      "Job 53ca24aa has completed for config 10\n",
      "Job d915518a has completed for config 4\n",
      "Job e426fcb2 has completed for config 13\n",
      "Job b2c8f344 has completed for config 8\n",
      "Job 97988df3 has completed for config 6\n",
      "Job 40c31d55 has completed for config 11\n",
      "Job 2d981cc1 has completed for config 15\n",
      "Job 010d4405 has completed for config 5\n",
      "Job 2721e218 has completed for config 12\n",
      "Job 01511755 has completed for config 2\n",
      "Job f3d8724e has completed for config 1\n",
      "Job 60964bcb has completed for config 3\n",
      "Job 63c1fcc8 has completed for config 0\n",
      "All 16 job(s) completed\n",
      "Config 0 produced metric 0.22 with params: {'hidden_layers_dim': 300, 'feedforward_const': 0.043951574155499704}\n",
      "Config 14 produced metric 0.31 with params: {'hidden_layers_dim': 300, 'feedforward_const': 0.05848771920419732}\n",
      "Config 6 produced metric 0.36 with params: {'hidden_layers_dim': 300, 'feedforward_const': 0.006299208962397572}\n",
      "Config 12 produced metric 0.44 with params: {'hidden_layers_dim': 100, 'feedforward_const': 0.05601180437954487}\n",
      "Config 4 produced metric 0.45 with params: {'hidden_layers_dim': 200, 'feedforward_const': 0.003246553415665837}\n",
      "Config 5 produced metric 0.57 with params: {'hidden_layers_dim': 200, 'feedforward_const': 0.0019433278974297264}\n",
      "Config 11 produced metric 0.61 with params: {'hidden_layers_dim': 100, 'feedforward_const': 0.0025973455297094625}\n",
      "Config 7 produced metric 1.14 with params: {'hidden_layers_dim': 100, 'feedforward_const': 0.00037441504751268363}\n",
      "Config 13 produced metric 1.26 with params: {'hidden_layers_dim': 200, 'feedforward_const': 0.00018638512254130432}\n",
      "Config 1 produced metric 1.43 with params: {'hidden_layers_dim': 300, 'feedforward_const': 0.00013025752255328113}\n",
      "Config 2 produced metric 88.76 with params: {'hidden_layers_dim': 200, 'feedforward_const': 5.414203512473023}\n",
      "Config 9 produced metric 88.76 with params: {'hidden_layers_dim': 100, 'feedforward_const': 0.7318551250004928}\n",
      "Config 10 produced metric 88.76 with params: {'hidden_layers_dim': 100, 'feedforward_const': 0.716883342271889}\n",
      "Config 15 produced metric 88.76 with params: {'hidden_layers_dim': 100, 'feedforward_const': 8.214786646778737}\n",
      "Config 3 produced metric 90.13 with params: {'hidden_layers_dim': 300, 'feedforward_const': 5.582105156163857}\n",
      "Config 8 produced metric 90.13 with params: {'hidden_layers_dim': 300, 'feedforward_const': 0.9893207844029632}\n"
     ]
    }
   ],
   "source": [
    "print(\"Submitting {0} jobs with {1} configurations \".format(str(num_configs), str(num_configs)))\n",
    "val_metric = queue.PriorityQueue()\n",
    "threads = []\n",
    "for index in job_configs:\n",
    "    parameter = generate_job_create_parameters(job_configs[index])\n",
    "    t = threading.Thread(\n",
    "        target=run_then_return_metric, \n",
    "        args = (index, resource_group, parameter, client, metric_extractor, val_metric))\n",
    "    threads.append(t)\n",
    "    t.daemon = True\n",
    "    t.start()\n",
    "\n",
    "for t in threads:\n",
    "    t.join()\n",
    "print(\"All {0} job(s) completed\".format(str(num_configs)))\n",
    "\n",
    "while not val_metric.empty():\n",
    "    metric, index = val_metric.get()\n",
    "    print(\"Config {0} produced metric {1} with params: {2}\".format(index, metric, job_configs[index]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete the Cluster\n",
    "When you are finished with the sample and don't want to submit any more jobs you can delete the cluster using the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "_ = client.clusters.delete(resource_group, cluster_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Delete File Share\n",
    "When you are finished with the sample and don't want to submit any more jobs you can delete the file share completely with all files using the following code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "service = FileService(storage_account_name, storage_account_key)\n",
    "service.delete_share(azure_file_share_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}