sdk/python/endpoints/online/kubernetes/kubernetes-online-endpoints-safe-rollout.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Safe rollout for online endpoints\n",
    "\n",
    "You've an existing model deployed in production and you want to deploy a new version of the model. How do you roll out your new machine learning model without causing any disruption? A good answer is blue-green deployment, an approach in which a new version of a web service is introduced to production by rolling out the change to a small subset of users/requests before rolling it out completely. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Requirements - In order to benefit from this tutorial, you will need:\n",
    "- This sample notebook assumes you're using online endpoints; for more information, see [What are Azure Machine Learning endpoints?](https://docs.microsoft.com/azure/machine-learning/concept-endpoints).\n",
    "- An Azure account with an active subscription. [Create an account for free](https://azure.microsoft.com/free/?WT.mc_id=A261C142F)\n",
    "- An Azure ML workspace. [Check this notebook for creating a workspace](/sdk/resources/workspace/workspace.ipynb)\n",
    "- Installed Azure Machine Learning Python SDK v2 - [install instructions](/sdk/README.md#getting-started)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### In this sample, you'll learn to:\n",
    "\n",
    "1. Deploy a new online endpoint called \"blue\" that serves version 1 of the model\n",
    "1. Scale this deployment so that it can handle more requests\n",
    "1. Deploy version 2 of the model to an endpoint called \"green\" that accepts no live traffic\n",
    "1. Test the green deployment in isolation\n",
    "1. Send 10% of live traffic to the green deployment\n",
    "1. Fully cut-over all live traffic to the green deployment\n",
    "1. Delete the now-unused v1 blue deployment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Connect to Azure Machine Learning Workspace\n",
    "The [workspace](https://docs.microsoft.com/azure/machine-learning/concept-workspace) is the top-level resource for Azure Machine Learning, providing a centralized place to work with all the artifacts you create when you use Azure Machine Learning. In this section we will connect to the workspace in which the job will be run."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.1 Import the required libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import required libraries\n",
    "from azure.ai.ml import MLClient\n",
    "from azure.ai.ml.entities import (\n",
    "    KubernetesOnlineEndpoint,\n",
    "    KubernetesOnlineDeployment,\n",
    "    Model,\n",
    "    Environment,\n",
    "    CodeConfiguration,\n",
    ")\n",
    "from azure.identity import DefaultAzureCredential\n",
    "from azure.ai.ml.entities._deployment.resource_requirements_settings import (\n",
    "    ResourceRequirementsSettings,\n",
    ")\n",
    "from azure.ai.ml.entities._deployment.container_resource_settings import (\n",
    "    ResourceSettings,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1.2 Configure workspace details and get a handle to the workspace\n",
    "\n",
    "To connect to a workspace, we need identifier parameters - a subscription, resource group and workspace name. We will use these details in the `MLClient` from `azure.ai.ml` to get a handle to the required Azure Machine Learning workspace. We use the default [interactive authentication](https://docs.microsoft.com/python/api/azure-identity/azure.identity.interactivebrowsercredential?view=azure-python) for this tutorial. More advanced connection methods can be found [here](https://docs.microsoft.com/python/api/azure-identity/azure.identity?view=azure-python)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# enter details of your AML workspace\n",
    "subscription_id = \"<SUBSCRIPTION_ID>\"\n",
    "resource_group = \"<RESOURCE_GROUP>\"\n",
    "workspace = \"<AML_WORKSPACE_NAME>\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get a handle to the workspace\n",
    "ml_client = MLClient(\n",
    "    DefaultAzureCredential(), subscription_id, resource_group, workspace\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Configure Kubernetes cluster for machine learning\n",
    "Next, configure Azure Kubernetes Service (AKS) and Azure Arc-enabled Kubernetes clusters for inferencing machine learning workloads.\n",
    "There're some prerequisites for below steps, you can check them [here](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-attach-arc-kubernetes).\n",
    "\n",
    "## 2.1 Connect an existing Kubernetes cluster to Azure Arc\n",
    "This step is optional for [AKS cluster](https://docs.microsoft.com/en-us/azure/aks/kubernetes-walkthrough).\n",
    "Follow this [guidance](https://docs.microsoft.com/en-us/azure/azure-arc/kubernetes/quickstart-connect-cluster) to connect Kubernetes clusters.\n",
    "\n",
    "## 2.2 Deploy Azure Machine Learning extension\n",
    "Depending on your network setup, Kubernetes distribution variant, and where your Kubernetes cluster is hosted (on-premises or the cloud), choose one of options to deploy the Azure Machine Learning extension and enable inferencing workloads on your Kubernetes cluster.\n",
    "Follow this [guidance](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-attach-arc-kubernetes?tabs=studio#inferencing).\n",
    "\n",
    "## 2.3 Attach Arc Cluster\n",
    "You can use Studio, Python SDK and CLI to attach Arc cluster to Machine Learning workspace.\n",
    "Below code shows the attachment of AKS that the compute type is `managedClusters`. For Arc connected cluster, it should be `connectedClusters`.\n",
    "Follow this [guidance](https://docs.microsoft.com/en-us/azure/machine-learning/how-to-attach-arc-kubernetes?tabs=studio#attach-arc-cluster) for more details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from azure.ai.ml import load_compute\n",
    "\n",
    "# for arc connected cluster, the resource_id should be something like '/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP>/providers/Microsoft.ContainerService/connectedClusters/<CLUSTER_NAME>''\n",
    "compute_params = [\n",
    "    {\"name\": \"<COMPUTE_NAME>\"},\n",
    "    {\"type\": \"kubernetes\"},\n",
    "    {\n",
    "        \"resource_id\": \"/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP>/providers/Microsoft.ContainerService/managedClusters/<CLUSTER_NAME>\"\n",
    "    },\n",
    "]\n",
    "k8s_compute = load_compute(source=None, params_override=compute_params)\n",
    "\n",
    "ml_client.begin_create_or_update(k8s_compute).result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Create Online Endpoint\n",
    "\n",
    "Online endpoints are endpoints that are used for online (real-time) inferencing. Online endpoints contain deployments that are ready to receive data from clients and can send responses back in real time.\n",
    "\n",
    "To create an online endpoint we will use `KubernetesOnlineEndpoint`. This class allows user to configure the following key aspects:\n",
    "\n",
    "- `name` - Name of the endpoint. Needs to be unique at the Azure region level\n",
    "- `auth_mode` - The authentication method for the endpoint. Key-based authentication and Azure ML token-based authentication are supported. Key-based authentication doesn't expire but Azure ML token-based authentication does. Possible values are `key` or `aml_token`.\n",
    "- `identity`- The managed identity configuration for accessing Azure resources for endpoint provisioning and inference.\n",
    "    - `type`- The type of managed identity. Azure Machine Learning supports `system_assigned` or `user_assigned identity`.\n",
    "    - `user_assigned_identities` - List (array) of fully qualified resource IDs of the user-assigned identities. This property is required is `identity.type` is user_assigned.\n",
    "- `description`- Description of the endpoint."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 Configure the endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating a unique endpoint name with current datetime to avoid conflicts\n",
    "import datetime\n",
    "\n",
    "online_endpoint_name = \"k8s-endpoint-\" + datetime.datetime.now().strftime(\"%m%d%H%M%f\")\n",
    "\n",
    "# create an online endpoint\n",
    "endpoint = KubernetesOnlineEndpoint(\n",
    "    name=online_endpoint_name,\n",
    "    compute=\"<COMPUTE_NAME>\",\n",
    "    description=\"this is a sample online endpoint\",\n",
    "    auth_mode=\"key\",\n",
    "    tags={\"foo\": \"bar\"},\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 Create the endpoint\n",
    "Using the `MLClient` created earlier, we will now create the Endpoint in the workspace. This command will start the endpoint creation and return a confirmation response while the endpoint creation continues."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml_client.begin_create_or_update(endpoint).result()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Create a blue deployment\n",
    "\n",
    "A deployment is a set of resources required for hosting the model that does the actual inferencing. We will create a deployment for our endpoint using the `KubernetesOnlineDeployment` class. This class allows user to configure the following key aspects.\n",
    "\n",
    "- `name` - Name of the deployment.\n",
    "- `endpoint_name` - Name of the endpoint to create the deployment under.\n",
    "- `model` - The model to use for the deployment. This value can be either a reference to an existing versioned model in the workspace or an inline model specification.\n",
    "- `environment` - The environment to use for the deployment. This value can be either a reference to an existing versioned environment in the workspace or an inline environment specification.\n",
    "- `code_configuration` - the configuration for the source code and scoring script\n",
    "    - `path`- Path to the source code directory for scoring the model\n",
    "    - `scoring_script` - Relative path to the scoring file in the source code directory\n",
    "- `instance_type` - The name of instance type for deployment to target certain types of nodes.\n",
    "- `instance_count` - The number of instances to use for the deployment\n",
    "- `resources` - The resource ask for the deployment with requests and limits.\n",
    "    - `requests` - The minimum resource ask for one deployment instance to be scheduled. For all deployment, requests for CPU and Memory should always be given.\n",
    "    - `limits` - (Optional) The maximum resource that one deployment instance can use. When values in limits missing, it will take the settings in instance type. For GPU workloads, limits for CPU and Memory should also be given. If specifying GPU for your deployment, it should be in field limits, or both requests and limits with the same value."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.1 Configure blue deployment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create blue deployment\n",
    "model = Model(path=\"../model-1/model/sklearn_regression_model.pkl\")\n",
    "env = Environment(\n",
    "    conda_file=\"../model-1/environment/conda.yml\",\n",
    "    image=\"mcr.microsoft.com/azureml/minimal-ubuntu18.04-py37-cpu-inference:latest\",\n",
    ")\n",
    "\n",
    "blue_deployment = KubernetesOnlineDeployment(\n",
    "    name=\"blue\",\n",
    "    endpoint_name=online_endpoint_name,\n",
    "    model=model,\n",
    "    environment=env,\n",
    "    code_configuration=CodeConfiguration(\n",
    "        code=\"../model-1/onlinescoring\", scoring_script=\"score.py\"\n",
    "    ),\n",
    "    instance_count=1,\n",
    "    resources=ResourceRequirementsSettings(\n",
    "        requests=ResourceSettings(\n",
    "            cpu=\"100m\",\n",
    "            memory=\"0.5Gi\",\n",
    "        ),\n",
    "    ),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4.2 Create the deployment\n",
    "\n",
    "Using the `MLClient` created earlier, we will now create the deployment in the workspace. This command will start the deployment creation and return a confirmation response while the deployment creation continues."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml_client.begin_create_or_update(blue_deployment).result()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# blue deployment takes 100 traffic\n",
    "endpoint.traffic = {\"blue\": 100}\n",
    "ml_client.begin_create_or_update(endpoint).result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5. Test the endpoint with sample data\n",
    "\n",
    "Using the `MLClient` created earlier, we will get a handle to the endpoint. The endpoint can be invoked using the invoke command with the following parameters:\n",
    "\n",
    "- `endpoint_name` - Name of the endpoint\n",
    "- `request_file` - File with request data\n",
    "- `deployment_name` - Name of the specific deployment to test in an endpoint\n",
    "\n",
    "We will send a sample request using a [json](./model-1/sample-request.json) file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# test the blue deployment with some sample data\n",
    "# comment this out as cluster under dev subscription can't be accessed from public internet.\n",
    "# ml_client.online_endpoints.invoke(\n",
    "#    endpoint_name=online_endpoint_name,\n",
    "#    deployment_name='blue',\n",
    "#    request_file='../model-1/sample-request.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 6. Scale the deployment\n",
    "\n",
    "Using the `MLClient` created earlier, we will get a handle to the deployment. The deployment can be scaled by increasing or decreasing the `instance count`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# scale the deployment\n",
    "blue_deployment = ml_client.online_deployments.get(\n",
    "    name=\"blue\", endpoint_name=online_endpoint_name\n",
    ")\n",
    "blue_deployment.instance_count = 2\n",
    "\n",
    "#!!!bug https://msdata.visualstudio.com/Vienna/_workitems/edit/1740434\n",
    "ml_client.online_deployments.begin_create_or_update(blue_deployment).result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 7. Get endpoint details"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the details for online endpoint\n",
    "endpoint = ml_client.online_endpoints.get(name=online_endpoint_name)\n",
    "\n",
    "# existing traffic details\n",
    "print(endpoint.traffic)\n",
    "\n",
    "# Get the scoring URI\n",
    "print(endpoint.scoring_uri)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 8. Deploy a new model, but send no traffic yet\n",
    "Create a new deployment named green"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create green deployment\n",
    "model2 = Model(path=\"../model-2/model/sklearn_regression_model.pkl\")\n",
    "env2 = Environment(\n",
    "    conda_file=\"../model-2/environment/conda.yml\",\n",
    "    image=\"mcr.microsoft.com/azureml/minimal-ubuntu18.04-py37-cpu-inference:latest\",\n",
    ")\n",
    "\n",
    "green_deployment = KubernetesOnlineDeployment(\n",
    "    name=\"green\",\n",
    "    endpoint_name=online_endpoint_name,\n",
    "    model=model2,\n",
    "    environment=env2,\n",
    "    code_configuration=CodeConfiguration(\n",
    "        code=\"../model-2/onlinescoring\", scoring_script=\"score.py\"\n",
    "    ),\n",
    "    instance_count=1,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use MLClient to create green deployment\n",
    "ml_client.begin_create_or_update(green_deployment).result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 9. Test green deployment \n",
    "Though green has 0% of traffic allocated, you can still invoke the endpoint and deployment with [json](./model-2/sample-request.json) file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# comment this out as cluster under dev subscription can't be accessed from public internet.\n",
    "# ml_client.online_endpoints.invoke(\n",
    "#    endpoint_name=online_endpoint_name,\n",
    "#    deployment_name='green',\n",
    "#    request_file='../model-2/sample-request.json')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9.1 Test the new deployment with a small percentage of live traffic\n",
    "Once you've tested your `green` deployment, allocate a small percentage of traffic to it:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint.traffic = {\"blue\": 90, \"green\": 10}\n",
    "ml_client.begin_create_or_update(endpoint).result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, your green deployment will receive 10% of requests."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 9.2 Send all traffic to your new deployment\n",
    "Once you're satisfied that your green deployment is fully satisfactory, switch all traffic to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "endpoint.traffic = {\"blue\": 0, \"green\": 100}\n",
    "ml_client.begin_create_or_update(endpoint).result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 10. Remove the old deployment\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml_client.online_deployments.begin_delete(\n",
    "    name=\"blue\", endpoint_name=online_endpoint_name\n",
    ").result()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 11. Delete endpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ml_client.online_endpoints.begin_delete(name=online_endpoint_name)"
   ]
  }
 ],
 "metadata": {
  "description": {
   "description": "Safely rollout a new version of a web service to production by rolling out the change to a small subset of users/requests before rolling it out completely"
  },
  "interpreter": {
   "hash": "fd3e9bb70c21fb1be2a8daaefc9841241bb4de49fe0d2fb8d8b5faf765ee3be6"
  },
  "kernelspec": {
   "display_name": "Python 3.10 - SDK V2",
   "language": "python",
   "name": "python310-sdkv2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}