01_Setup_AML_Env.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 01. Azure Machine Learning - Environment Setup\n",
    "This notebook gets a reference to an Azure Machine Learning workspace using the AML SDK and performs a few operations in preparation for this demo. \n",
    " - Create a compute cluster to be used for model training\n",
    " - Upload sample weld images to a new blob datastore. \n",
    " - Creates a labeled image dataset (using existing labels) that can be used as an input to an AutoML for Images training job\n",
    " \n",
    "Run all of the cells in this notebook to setup the necessary components for the ML experiment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import required packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azureml.core import Workspace, Datastore, Dataset\n",
    "from azureml.data import DataType\n",
    "from azureml.core.compute import ComputeTarget, AmlCompute\n",
    "from azureml.core.compute_target import ComputeTargetException"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Connect to AML workspace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ws = Workspace.from_config()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create Azure ML Compute Cluster for model training\n",
    "Compute clusters scalable, on-demand, compute resources that can be dynamically spun up and down to support different model training and inferencing jobs - think of compute clusters as massively scalable, serverless, ML compute. Here, we are creating a GPU cluster that can provision up to 5 nodes depending upon the workload submitted - this upper limit can be increased even futher. Once jobs complete and nodes sit idle for 120 seconds they are automatically spun down and billing stops.\n",
    "\n",
    "[Create an Azure Machine Learning Compute Cluster](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-create-attach-compute-cluster?tabs=python)\n",
    "\n",
    "[What are Compute Targets in Azure Machine Learning](https://docs.microsoft.com/en-us/azure/machine-learning/concept-compute-target)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cluster_name = 'automlimagescompute'\n",
    "# compute_target = ws.compute_targets[cluster_name]\n",
    "try:\n",
    "    compute_target = ComputeTarget(workspace=ws, name=cluster_name)\n",
    "    print(\"Found existing compute target.\")\n",
    "except ComputeTargetException:\n",
    "    print(\"Creating a new compute target...\")\n",
    "    compute_config = AmlCompute.provisioning_configuration(\n",
    "        vm_size='Standard_NC6s_v3',\n",
    "        location='westus3',\n",
    "        idle_seconds_before_scaledown=120,\n",
    "        min_nodes=0,\n",
    "        max_nodes=5,\n",
    "    )\n",
    "    compute_target = ComputeTarget.create(ws, cluster_name, compute_config)\n",
    "    # Can poll for a minimum number of nodes and for a specific timeout.\n",
    "    # If no min_node_count is provided, it will use the scale settings for the cluster.\n",
    "    compute_target.wait_for_completion(\n",
    "        show_output=True, min_node_count=None, timeout_in_minutes=20\n",
    "    )\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Get pointer to default Azure ML Blob Datastore\n",
    "Datastores represent storage locations that are attached to an Azure Machine Learning Workspace. Connection information for these storage services are stored in an AML-linked Key Vault and they can be accessed either through the AML UI or SDK.\n",
    "\n",
    "Azure Machine Learning workspaces are provisioned with an attached Azure Storage Account by default. These default blobstores are generally used for storing artifacts from ML experiments like trained models, saved outputs, run logs etc. Here we retrieve a reference to the default datastore.\n",
    "\n",
    "[Connect to Storage Services on Azure Machine Learning with Datastores](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-access-data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "default_ds = ws.get_default_datastore()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create new datastore and upload sample images\n",
    "Azure Blob Storage containers can be registered as datastores in an AML workspace. For the purposes of our demonstration, we are going to create a separate blob container (in the attached default storage account) specifically for image storage, and register this as a new datastore. <i>Note:</i> This is purely for demonstration purposes, images could be loaded from any target datastore for model training.\n",
    "\n",
    "[Creating a New Azure Machine Learning Datastore]('https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.core.datastore.datastore?view=azure-ml-py')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "new_container_name = 'streetimagestore'\n",
    "default_ds.blob_service.create_container(new_container_name)\n",
    "\n",
    "try:\n",
    "    imagestore = Datastore.get(ws, new_container_name)\n",
    "    print(\"Found Blob Datastore with name: %s\" % new_container_name)\n",
    "except Exception:\n",
    "    imagestore = Datastore.register_azure_blob_container(\n",
    "     workspace=ws,\n",
    "        datastore_name = new_container_name, \n",
    "        account_name = default_ds.account_name,\n",
    "        container_name = new_container_name,\n",
    "        account_key = default_ds.account_key)\n",
    "    print(\"Registered blob datastore with name: %s\" % new_container_name)\n",
    "\n",
    "# Upload images to new datastore\n",
    "imagestore.upload('./sample_images', 'aml_images', overwrite=True, show_progress=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create labeled Dataset in the Azure Machine Learning workspace\n",
    "To train a custom instance segmentation model we need to provide a labeled dataset to the AutoML for Images job. For the purposes of our demo we're utilizing pre-labeled a dataset of street images images from the [CBCL StreetScenes Challenge Framework](http://cbcl.mit.edu/software-datasets/streetscenes/) (see `./saml_annotations/*.jsonl`). The code below will upload two labeled datasets and register them in the AML workspace as `TRAIN_AML_Labeled_Street_Images`  and `TEST_AML_Labeled_Street_Images`.\n",
    "\n",
    "[Labeling Images and Text Documents in Azure ML](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-label-data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "default_ds.upload('./aml_annotations', 'labeled_image_files', overwrite=True, show_progress=True)\n",
    "\n",
    "train_image_dataset = Dataset.Tabular.from_json_lines_files(path=(default_ds, 'labeled_image_files/labeled_images_train.jsonl'), set_column_types={\"image_url\": DataType.to_stream(imagestore.workspace)})\n",
    "train_image_dataset.register(ws, 'TRAIN_AML_Labeled_Street_Images', create_new_version=True)\n",
    "\n",
    "test_image_dataset = Dataset.Tabular.from_json_lines_files(path=(default_ds, 'labeled_image_files/labeled_images_test.jsonl'), set_column_types={\"image_url\": DataType.to_stream(imagestore.workspace)})\n",
    "test_image_dataset.register(ws, 'TEST_AML_Labeled_Street_Images', create_new_version=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## AML Environment setup complete!\n",
    "\n",
    "Your Azure ML Workspace has been populated with sample weld images, a registered dataset, and you have available GPU compute ready to support model training operations!"
   ]
  }
 ],
 "metadata": {
  "kernel_info": {
   "name": "python38-azureml"
  },
  "kernelspec": {
   "display_name": "Python 3.8 - AzureML",
   "language": "python",
   "name": "python38-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "nteract": {
   "version": "nteract-front-end@1.0.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}