04 - Run Experiments.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Run Experiments\n",
        "\n",
        "You can use the Azure Machine Learning SDK to run code experiments that log metrics and generate outputs. This is at the core of most machine learning operations in Azure Machine Learning.\n",
        "\n",
        "## Connect to your workspace\n",
        "\n",
        "All experiments and associated resources are managed within your Azure Machine Learning workspace. In most cases, you should store the workspace configuration in a JSON configuration file. This makes it easier to reconnect without needing to remember details like your Azure subscription ID. You can download the JSON configuration file from the blade for your workspace in the Azure portal, but if you're using a Compute Instance within your workspace, the configuration file has already been downloaded to the root folder.\n",
        "\n",
        "The code below uses the configuration file to connect to your workspace.\n",
        "\n",
        "> **Note**: If you haven't already established an authenticated session with your Azure subscription, you'll be prompted to authenticate by clicking a link, entering an authentication code, and signing into Azure."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649363615859
        }
      },
      "outputs": [],
      "source": [
        "import azureml.core\n",
        "from azureml.core import Workspace\n",
        "\n",
        "# Load the workspace from the saved config file\n",
        "ws = Workspace.from_config()\n",
        "print('Ready to use Azure ML {} to work with {}'.format(azureml.core.VERSION, ws.name))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Run an experiment\n",
        "\n",
        "One of the most fundamental tasks that data scientists need to perform is to create and run experiments that process and analyze data. In this exercise, you'll learn how to use an Azure ML *experiment* to run Python code and record values extracted from data. In this case, you'll use a simple dataset that contains details of patients that have been tested for diabetes. You'll run an experiment to explore the data, extracting statistics, visualizations, and data samples. Most of the code you'll use is fairly generic Python, such as you might run in any data exploration process. However, with the addition of a few lines, the code uses an Azure ML *experiment* to log details of the run."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649363619696
        },
        "scrolled": false
      },
      "outputs": [],
      "source": [
        "from azureml.core import Experiment\n",
        "import pandas as pd\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline \n",
        "\n",
        "# Create an Azure ML experiment in your workspace\n",
        "experiment = Experiment(workspace=ws, name=\"mslearn-diabetes\")\n",
        "\n",
        "# Start logging data from the experiment, obtaining a reference to the experiment run\n",
        "run = experiment.start_logging()\n",
        "print(\"Starting experiment:\", experiment.name)\n",
        "\n",
        "# load the data from a local file\n",
        "data = pd.read_csv('data/diabetes.csv')\n",
        "\n",
        "# Count the rows and log the result\n",
        "row_count = (len(data))\n",
        "run.log('observations', row_count)\n",
        "print('Analyzing {} rows of data'.format(row_count))\n",
        "\n",
        "# Plot and log the count of diabetic vs non-diabetic patients\n",
        "diabetic_counts = data['Diabetic'].value_counts()\n",
        "fig = plt.figure(figsize=(6,6))\n",
        "ax = fig.gca()    \n",
        "diabetic_counts.plot.bar(ax = ax) \n",
        "ax.set_title('Patients with Diabetes') \n",
        "ax.set_xlabel('Diagnosis') \n",
        "ax.set_ylabel('Patients')\n",
        "plt.show()\n",
        "run.log_image(name='label distribution', plot=fig)\n",
        "\n",
        "# log distinct pregnancy counts\n",
        "pregnancies = data.Pregnancies.unique()\n",
        "run.log_list('pregnancy categories', pregnancies)\n",
        "\n",
        "# Log summary statistics for numeric columns\n",
        "med_columns = ['PlasmaGlucose', 'DiastolicBloodPressure', 'TricepsThickness', 'SerumInsulin', 'BMI']\n",
        "summary_stats = data[med_columns].describe().to_dict()\n",
        "for col in summary_stats:\n",
        "    keys = list(summary_stats[col].keys())\n",
        "    values = list(summary_stats[col].values())\n",
        "    for index in range(len(keys)):\n",
        "        run.log_row(col, stat=keys[index], value = values[index])\n",
        "        \n",
        "# Save a sample of the data and upload it to the experiment output\n",
        "data.sample(100).to_csv('sample.csv', index=False, header=True)\n",
        "run.upload_file(name='outputs/sample.csv', path_or_stream='./sample.csv')\n",
        "\n",
        "# Complete the run\n",
        "run.complete()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## View run details\n",
        "\n",
        "In Jupyter Notebooks, you can use the **RunDetails** widget to see a visualization of the run details."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649363626246
        }
      },
      "outputs": [],
      "source": [
        "from azureml.widgets import RunDetails\n",
        "\n",
        "RunDetails(run).show()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### View more details in Azure Machine Learning studio\n",
        "\n",
        "Note that the **RunDetails** widget includes a link to **view run details** in Azure Machine Learning studio. Click this to open a new browser tab with the run details (you can also just open [Azure Machine Learning studio](https://ml.azure.com) and find the run on the **Jobs** page). When viewing the job run in Azure Machine Learning studio, note the following:\n",
        "\n",
        "- The **Overview** tab contains the general properties of the experiment run.\n",
        "- The **Metrics** tab enables you to select logged metrics and view them as tables or charts.\n",
        "- The **Images** tab enables you to select and view any images or plots that were logged in the experiment (in this case, the *Label Distribution* plot)\n",
        "- The **Child jobs** tab lists any child runs (in this experiment there are none).\n",
        "- The **Outputs + logs** tab shows the output or log files generated by the experiment.\n",
        "- The **Code** tab contains all files in the folder where the experiment code was run (in this case, everything in the same folder as this notebook).\n",
        "- The **Explanations** tab is used to show model explanations generated by the experiment (in this case, there are none).\n",
        "- The **Fairness** tab is used to visualize predictive performance disparities that help you evaluate the fairness of machine learning models (in this case, there are none)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Retrieve experiment details using the SDK\n",
        "\n",
        "The **run** variable in the code you ran previously is an instance of a **Run** object, which is a reference to an individual run of an experiment in Azure Machine Learning. You can use this reference to get information about the run and its outputs:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649363632957
        }
      },
      "outputs": [],
      "source": [
        "import json\n",
        "\n",
        "# Get logged metrics\n",
        "print(\"Metrics:\")\n",
        "metrics = run.get_metrics()\n",
        "for metric_name in metrics:\n",
        "    print(metric_name, \":\", metrics[metric_name])\n",
        "\n",
        "# Get output files\n",
        "print(\"\\nFiles:\")\n",
        "files = run.get_file_names()\n",
        "for file in files:\n",
        "    print(file)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "You can download the files produced by the experiment, either individually by using the **download_file** method, or by using the **download_files** method to retrieve multiple files. The following code downloads all of the files in the run's **output** folder:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649363635892
        }
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "download_folder = 'downloaded-files'\n",
        "\n",
        "# Download files in the \"outputs\" folder\n",
        "run.download_files(prefix='outputs', output_directory=download_folder)\n",
        "\n",
        "# Verify the files have been downloaded\n",
        "for root, directories, filenames in os.walk(download_folder): \n",
        "    for filename in filenames:  \n",
        "        print (os.path.join(root,filename))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "If you need to troubleshoot the experiment run, you can use the **get_details** method to retrieve basic details about the run, or you can use the **get_details_with_logs** method to retrieve the run details as well as the contents of log files generated during the run:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649363638967
        }
      },
      "outputs": [],
      "source": [
        "run.get_details_with_logs()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Note that the details include information about the compute target on which the experiment was run, the date and time when it started and ended. Additionally, because the notebook containing the experiment code (this one) is in a cloned Git repository, details about the repo, branch, and status are recorded in the run history.\n",
        "\n",
        "In this case, note that the **logFiles** entry in the details indicates that no log files were generated. That's typical for an inline experiment like the one you ran, but things get more interesting when you run a script as an experiment; which is what we'll look at next."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Run an experiment script\n",
        "\n",
        "In the previous example, you ran an experiment inline in this notebook. A more flexible solution is to create a separate script for the experiment, and store it in a folder along with any other files it needs, and then use Azure ML to run the experiment based on the script in the folder.\n",
        "\n",
        "First, let's create a folder for the experiment files, and copy the data into it:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649363641423
        }
      },
      "outputs": [],
      "source": [
        "import os, shutil\n",
        "\n",
        "# Create a folder for the experiment files\n",
        "folder_name = 'diabetes-experiment-files'\n",
        "experiment_folder = './' + folder_name\n",
        "os.makedirs(folder_name, exist_ok=True)\n",
        "\n",
        "# Copy the data file into the experiment folder\n",
        "shutil.copy('data/diabetes.csv', os.path.join(folder_name, \"diabetes.csv\"))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now we'll create a Python script containing the code for our experiment, and save it in the experiment folder.\n",
        "\n",
        "> **Note**: running the following cell just *creates* the script file - it doesn't run it!"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%writefile $folder_name/diabetes_experiment.py\n",
        "from azureml.core import Run\n",
        "import pandas as pd\n",
        "import os\n",
        "\n",
        "# Get the experiment run context\n",
        "run = Run.get_context()\n",
        "\n",
        "# load the diabetes dataset\n",
        "data = pd.read_csv('diabetes.csv')\n",
        "\n",
        "# Count the rows and log the result\n",
        "row_count = (len(data))\n",
        "run.log('observations', row_count)\n",
        "print('Analyzing {} rows of data'.format(row_count))\n",
        "\n",
        "# Count and log the label counts\n",
        "diabetic_counts = data['Diabetic'].value_counts()\n",
        "print(diabetic_counts)\n",
        "for k, v in diabetic_counts.items():\n",
        "    run.log('Label:' + str(k), v)\n",
        "      \n",
        "# Save a sample of the data in the outputs folder (which gets uploaded automatically)\n",
        "os.makedirs('outputs', exist_ok=True)\n",
        "data.sample(100).to_csv(\"outputs/sample.csv\", index=False, header=True)\n",
        "\n",
        "# Complete the run\n",
        "run.complete()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "This code is a simplified version of the inline code used before. However, note the following:\n",
        "- It uses the `Run.get_context()` method to retrieve the experiment run context when the script is run.\n",
        "- It loads the diabetes data from the folder where the script is located.\n",
        "- It creates a folder named **outputs** and writes the sample file to it - this folder is automatically uploaded to the experiment run"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now you're almost ready to run the experiment. To run the script, you must create a **ScriptRunConfig** that identifies the Python script file to be run in the experiment, and then run an experiment based on it.\n",
        "\n",
        "> **Note**: The ScriptRunConfig also determines the compute target and Python environment. In this case, the Python environment is defined to include some Conda and pip packages, but the compute target is omitted; so the default local compute will be used.\n",
        "\n",
        "The following cell configures and submits the script-based experiment."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649363661425
        }
      },
      "outputs": [],
      "source": [
        "from azureml.core import Experiment, ScriptRunConfig, Environment\n",
        "from azureml.core.runconfig import DockerConfiguration\n",
        "from azureml.widgets import RunDetails\n",
        "\n",
        "# Create a Python environment for the experiment (from a .yml file)\n",
        "env = Environment.from_conda_specification(\"experiment_env\", \"environment.yml\")\n",
        "\n",
        "# Create a script config\n",
        "script_config = ScriptRunConfig(source_directory=experiment_folder,\n",
        "                                script='diabetes_experiment.py',\n",
        "                                environment=env,\n",
        "                                docker_runtime_config=DockerConfiguration(use_docker=True))\n",
        "\n",
        "# submit the experiment\n",
        "experiment = Experiment(workspace=ws, name='mslearn-diabetes')\n",
        "run = experiment.submit(config=script_config)\n",
        "RunDetails(run).show()\n",
        "run.wait_for_completion()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "As before, you can use the widget or the link to the experiment in [Azure Machine Learning studio](https://ml.azure.com) to view the outputs generated by the experiment, and you can also write code to retrieve the metrics and files it generated:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649363665408
        }
      },
      "outputs": [],
      "source": [
        "# Get logged metrics\n",
        "metrics = run.get_metrics()\n",
        "for key in metrics.keys():\n",
        "        print(key, metrics.get(key))\n",
        "print('\\n')\n",
        "for file in run.get_file_names():\n",
        "    print(file)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Note that this time, the run generated some log files. You can view these in the widget, or you can use the **get_details_with_logs** method like we did before, only this time the output will include the log data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649363668206
        }
      },
      "outputs": [],
      "source": [
        "run.get_details_with_logs()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Although you can view the log details in the output above, it's usually easier to download the log files and view them in a text editor."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649363670778
        }
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "log_folder = 'downloaded-logs'\n",
        "\n",
        "# Download all files\n",
        "run.get_all_logs(destination=log_folder)\n",
        "\n",
        "# Verify the files have been downloaded\n",
        "for root, directories, filenames in os.walk(log_folder): \n",
        "    for filename in filenames:  \n",
        "        print (os.path.join(root,filename))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## View experiment run history\n",
        "\n",
        "Now that you've run the same experiment multiple times, you can view the history in [Azure Machine Learning studio](https://ml.azure.com) and explore each logged run. Or you can retrieve an experiment by name from the workspace and iterate through its runs using the SDK:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649363706798
        }
      },
      "outputs": [],
      "source": [
        "from azureml.core import Experiment, Run\n",
        "\n",
        "diabetes_experiment = ws.experiments['mslearn-diabetes']\n",
        "for logged_run in diabetes_experiment.get_runs():\n",
        "    print('Run ID:', logged_run.id)\n",
        "    metrics = logged_run.get_metrics()\n",
        "    for key in metrics.keys():\n",
        "        print('-', key, metrics.get(key))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Use MLflow\n",
        "\n",
        "MLflow is an open source platform for managing machine learning processes. It's commonly (but not exclusively) used in Databricks environments to coordinate experiments and track metrics. In Azure Machine Learning experiments, you can use MLflow to track metrics as an alternative to the native log functionality.\n",
        "\n",
        "To take advantage of this capability, you'll need the **azureml-mlflow** package, so let's ensure it's installed."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649363713535
        }
      },
      "outputs": [],
      "source": [
        "pip show azureml-mlflow"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "### Use MLflow with an inline experiment\n",
        "\n",
        "To use MLflow to track metrics for an inline experiment, you must set the MLflow *tracking URI* to the workspace where the experiment is being run. This enables you to use **mlflow** tracking methods to log data to the experiment run."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649363717485
        }
      },
      "outputs": [],
      "source": [
        "from azureml.core import Experiment\n",
        "import pandas as pd\n",
        "import mlflow\n",
        "\n",
        "# Set the MLflow tracking URI to the workspace\n",
        "mlflow.set_tracking_uri(ws.get_mlflow_tracking_uri())\n",
        "\n",
        "# Create an Azure ML experiment in your workspace\n",
        "experiment = Experiment(workspace=ws, name='mslearn-diabetes-mlflow')\n",
        "mlflow.set_experiment(experiment.name)\n",
        "\n",
        "# start the MLflow experiment\n",
        "with mlflow.start_run():\n",
        "    \n",
        "    print(\"Starting experiment:\", experiment.name)\n",
        "    \n",
        "    # Load data\n",
        "    data = pd.read_csv('data/diabetes.csv')\n",
        "\n",
        "    # Count the rows and log the result\n",
        "    row_count = (len(data))\n",
        "    mlflow.log_metric('observations', row_count)\n",
        "    print(\"Run complete\")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Now let's look at the metrics logged during the run"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649363720202
        }
      },
      "outputs": [],
      "source": [
        "# Get the latest run of the experiment\n",
        "run = list(experiment.get_runs())[0]\n",
        "\n",
        "# Get logged metrics\n",
        "print(\"\\nMetrics:\")\n",
        "metrics = run.get_metrics()\n",
        "for key in metrics.keys():\n",
        "        print(key, metrics.get(key))\n",
        "    \n",
        "# Get a link to the experiment in Azure ML studio   \n",
        "experiment_url = experiment.get_portal_url()\n",
        "print('See details at', experiment_url)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "After running the code above, you can use the link that is displayed to view the experiment in Azure Machine Learning studio. Then select the latest run of the experiment and view its **Metrics** tab to see the logged metric.\n",
        "\n",
        "### Use MLflow in an experiment script\n",
        "\n",
        "You can also use MLflow to track metrics in an experiment script.\n",
        "\n",
        "Run the following two cells to create a folder and a script for an experiment that uses MLflow."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649363722917
        }
      },
      "outputs": [],
      "source": [
        "import os, shutil\n",
        "\n",
        "# Create a folder for the experiment files\n",
        "folder_name = 'mlflow-experiment-files'\n",
        "experiment_folder = './' + folder_name\n",
        "os.makedirs(folder_name, exist_ok=True)\n",
        "\n",
        "# Copy the data file into the experiment folder\n",
        "shutil.copy('data/diabetes.csv', os.path.join(folder_name, \"diabetes.csv\"))"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "%%writefile $folder_name/mlflow_diabetes.py\n",
        "from azureml.core import Run\n",
        "import pandas as pd\n",
        "import mlflow\n",
        "\n",
        "\n",
        "# start the MLflow experiment\n",
        "with mlflow.start_run():\n",
        "       \n",
        "    # Load data\n",
        "    data = pd.read_csv('diabetes.csv')\n",
        "\n",
        "    # Count the rows and log the result\n",
        "    row_count = (len(data))\n",
        "    print('observations:', row_count)\n",
        "    mlflow.log_metric('observations', row_count)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "When you use MLflow tracking in an Azure ML experiment script, the MLflow tracking URI is set automatically when you start the experiment run. However, the environment in which the script is to be run must include the required **mlflow** packages."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649363740151
        },
        "scrolled": false
      },
      "outputs": [],
      "source": [
        "from azureml.core import Experiment, ScriptRunConfig, Environment\n",
        "from azureml.core.runconfig import DockerConfiguration\n",
        "from azureml.widgets import RunDetails\n",
        "\n",
        "\n",
        "# Create a Python environment for the experiment (from a .yml file)\n",
        "env = Environment.from_conda_specification(\"experiment_env\", \"environment.yml\")\n",
        "\n",
        "# Create a script config\n",
        "script_mlflow = ScriptRunConfig(source_directory=experiment_folder,\n",
        "                                script='mlflow_diabetes.py',\n",
        "                                environment=env,\n",
        "                                docker_runtime_config=DockerConfiguration(use_docker=True)) \n",
        "\n",
        "# submit the experiment\n",
        "experiment = Experiment(workspace=ws, name='mslearn-diabetes-mlflow')\n",
        "run = experiment.submit(config=script_mlflow)\n",
        "RunDetails(run).show()\n",
        "run.wait_for_completion()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "As usual, you can get the logged metrics from the experiment run when it's finished."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "gather": {
          "logged": 1649363744091
        }
      },
      "outputs": [],
      "source": [
        "# Get logged metrics\n",
        "metrics = run.get_metrics()\n",
        "for key in metrics.keys():\n",
        "        print(key, metrics.get(key))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "> **More Information**: To find out more about running experiments, see [this topic](https://docs.microsoft.com/azure/machine-learning/how-to-manage-runs) in the Azure ML documentation. For details of how to log metrics in a run, see [this topic](https://docs.microsoft.com/azure/machine-learning/how-to-track-experiments). For more information about integrating Azure ML experiments with MLflow, see [this topic](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-use-mlflow)."
      ]
    }
  ],
  "metadata": {
    "kernel_info": {
      "name": "python38-azureml"
    },
    "kernelspec": {
      "display_name": "Python 3.9.5 ('base')",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.9.5"
    },
    "nteract": {
      "version": "nteract-front-end@1.0.0"
    },
    "vscode": {
      "interpreter": {
        "hash": "f2b2cd046deda8eabef1e765a11d0ec9aa9bd1d31d56ce79c815a38c323e14ec"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}