02_automated_pipeline_deploy.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "15dd06a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copyright 2022 Google LLC\n",
    "# Authors: \n",
    "# Fabian Hirschmann <fhirschmann@google.com>, \n",
    "# Elia Secchi <eliasecchi@google.com>,\n",
    "# Megha Agarwal <meghaag@google.com>,\n",
    "# Mandie Quartly <mandieq@google.com>\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d70205c",
   "metadata": {},
   "source": [
    "# Automated MLOps pipeline build, testing and deployment\n",
    "\n",
    "In the previous notebook, you created a machine learning pipeline to train a model. In this session, it's all about automating the training and deployment of this model. Hence, the objective this notebook is to:\n",
    "\n",
    "1. Refactor your Kubeflow pipeline into a Python file that can be compiled into YAML in an automated fashion.\n",
    "1. Write a script to deploy a compiled Kubeflow pipeline to Vertex AI.\n",
    "1. Use Cloud Build (CI/CD) to compile, test, and run your Kubeflow pipeline.\n",
    "1. Create a Cloud Source Repository (Git) to automatically trigger Cloud Build on every change on the master branch\n",
    "1. Create a [pipeline template](https://cloud.google.com/vertex-ai/docs/pipelines/create-pipeline-template) to allow for the pipeline to be reused and retriggered\n",
    "1. Setup a schedule for the pipeline, which can be done in 2 ways:\n",
    "      - using Cloud Scheduler job, which sends a message to a Pub/Sub topic, and then calls a Cloud Function to trigger the VertexAI pipeline.\n",
    "      - we anticipate the new Vertex AI Pipelines Schedules API\n",
    "      \n",
    "      \n",
    "As part of this notebook, you'll create the following files using the `%%writefile` directive.\n",
    "- `src/requirements.txt`: Python requirements file listing all dependencies.\n",
    "- `src/pipeline.py`: File containing your Kubeflow pipeline in Python and logic to compile the pipeline into YAML.\n",
    "- `src/create-pipeline-template.py`: Python script to create a Kubeflow Pipeline Template in Vertex AI.\n",
    "- `src/submit-pipeline.py`: Python script to submit pipeline to Vertex AI.\n",
    "- `src/cloudbuild.yml`: Cloud build pipeline to run your CI/CD process.\n",
    "- `src/tests/test_pipeline.py`: Unit test for pipeline\n",
    "- `cf-trigger/main.py`: Cloud Function to trigger your Kubeflow Pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e533bcd",
   "metadata": {},
   "source": [
    "## Prerequisites"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1ce3c4c-cd58-45aa-9e74-eb341a116ac8",
   "metadata": {},
   "source": [
    "Using the cell below, ensures that the following cloud service APIs are enabled for this lab:\n",
    "1. `Vertex AI API`\n",
    "1. `Cloud Build API`\n",
    "1. `Artifact Registry API`\n",
    "1. `Cloud Source Repositories API`\n",
    "1. `Cloud Function API`\n",
    "1. `Cloud Scheduler API`\n",
    "\n",
    "It also ensures that your Compute Engine default service account `{PROJECT_NUMBER}-compute@developer.gserviceaccount.com` has following permissions enabled:\n",
    "1. `Storage Admin`\n",
    "1. `Vertex AI User`\n",
    "1. `Cloud Build Editor`\n",
    "1. `Artifact Registry Writer`\n",
    "1. `Source Repository Administrator`\n",
    "\n",
    "And that your Cloud Build default service account `{PROJECT_NUMBER}-@cloudbuild.gserviceaccount.com` has following permissions enabled:\n",
    "1. `Service Account User`\n",
    "1. `Vertex AI User`\n",
    "\n",
    "If the credentials you are using in this notebooks don't have the right permissions to run the following cell, feel free to run these in [Cloud Shell](https://cloud.google.com/shell/docs/using-cloud-shell) to setup your environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e1c89940-1f9a-4241-817a-4165f3f35a23",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "gcloud services enable cloudbuild.googleapis.com\n",
    "gcloud services enable artifactregistry.googleapis.com\n",
    "gcloud services enable sourcerepo.googleapis.com\n",
    "gcloud services enable cloudfunctions.googleapis.com\n",
    "gcloud services enable cloudscheduler.googleapis.com\n",
    "\n",
    "PROJECT_ID=$(gcloud config get-value project)\n",
    "# Default Compute Engine SA roles\n",
    "PROJECT_NUM=$(gcloud projects list --filter=\"$PROJECT_ID\" --format=\"value(PROJECT_NUMBER)\")\n",
    "gcloud projects add-iam-policy-binding $PROJECT_ID \\\n",
    "      --member=\"serviceAccount:${PROJECT_NUM}-compute@developer.gserviceaccount.com\"\\\n",
    "      --role='roles/storage.admin'\n",
    "gcloud projects add-iam-policy-binding $PROJECT_ID \\\n",
    "      --member=\"serviceAccount:${PROJECT_NUM}-compute@developer.gserviceaccount.com\"\\\n",
    "      --role='roles/aiplatform.user'\n",
    "gcloud projects add-iam-policy-binding $PROJECT_ID \\\n",
    "      --member=\"serviceAccount:${PROJECT_NUM}-compute@developer.gserviceaccount.com\"\\\n",
    "      --role='roles/cloudbuild.builds.editor'\n",
    "gcloud projects add-iam-policy-binding $PROJECT_ID \\\n",
    "      --member=\"serviceAccount:${PROJECT_NUM}-compute@developer.gserviceaccount.com\"\\\n",
    "      --role='roles/artifactregistry.writer'\n",
    "gcloud projects add-iam-policy-binding $PROJECT_ID \\\n",
    "      --member=\"serviceAccount:${PROJECT_NUM}-compute@developer.gserviceaccount.com\"\\\n",
    "      --role='roles/source.admin'\n",
    "\n",
    "# Default Cloud Build SA roles\n",
    "gcloud projects add-iam-policy-binding $PROJECT_ID \\\n",
    "      --member=\"serviceAccount:${PROJECT_NUM}@cloudbuild.gserviceaccount.com\"\\\n",
    "      --role='roles/aiplatform.user'\n",
    "gcloud projects add-iam-policy-binding $PROJECT_ID \\\n",
    "      --member=\"serviceAccount:${PROJECT_NUM}@cloudbuild.gserviceaccount.com\"\\\n",
    "      --role='roles/iam.serviceAccountUser'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a48c4f38-2063-4c55-80c4-f2507e689f09",
   "metadata": {},
   "source": [
    "### Install required python packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "27fcba45",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p src"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "669e8ffe",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting src/requirements.txt\n"
     ]
    }
   ],
   "source": [
    "%%writefile src/requirements.txt\n",
    "kfp==2.0.0b12\n",
    "pytest==7.2.0\n",
    "pytz==2022.7\n",
    "google-cloud-aiplatform==1.20.0\n",
    "google-api-core==2.10.2\n",
    "google-auth==1.35.0\n",
    "google-cloud-bigquery==1.20.0\n",
    "google-cloud-core==1.7.3\n",
    "google-cloud-resource-manager==1.6.3\n",
    "google-cloud-storage==2.2.1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "bdf05ca5",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting kfp==2.0.0b12\n",
      "  Downloading kfp-2.0.0-beta.12.tar.gz (492 kB)\n",
      "\u001b[2K     \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m492.9/492.9 kB\u001b[0m \u001b[31m8.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0ma \u001b[36m0:00:01\u001b[0m\n",
      "\u001b[?25h  Preparing metadata (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25hRequirement already satisfied: pytest==7.2.0 in /home/jupyter/.local/lib/python3.7/site-packages (from -r src/requirements.txt (line 2)) (7.2.0)\n",
      "Requirement already satisfied: pytz==2022.7 in /home/jupyter/.local/lib/python3.7/site-packages (from -r src/requirements.txt (line 3)) (2022.7)\n",
      "Requirement already satisfied: google-cloud-aiplatform==1.20.0 in /home/jupyter/.local/lib/python3.7/site-packages (from -r src/requirements.txt (line 4)) (1.20.0)\n",
      "Requirement already satisfied: google-api-core==2.10.2 in /home/jupyter/.local/lib/python3.7/site-packages (from -r src/requirements.txt (line 5)) (2.10.2)\n",
      "Requirement already satisfied: google-auth==1.35.0 in /home/jupyter/.local/lib/python3.7/site-packages (from -r src/requirements.txt (line 6)) (1.35.0)\n",
      "Requirement already satisfied: google-cloud-bigquery==1.20.0 in /home/jupyter/.local/lib/python3.7/site-packages (from -r src/requirements.txt (line 7)) (1.20.0)\n",
      "Requirement already satisfied: google-cloud-core==1.7.3 in /home/jupyter/.local/lib/python3.7/site-packages (from -r src/requirements.txt (line 8)) (1.7.3)\n",
      "Requirement already satisfied: google-cloud-resource-manager==1.6.3 in /home/jupyter/.local/lib/python3.7/site-packages (from -r src/requirements.txt (line 9)) (1.6.3)\n",
      "Requirement already satisfied: google-cloud-storage==2.2.1 in /home/jupyter/.local/lib/python3.7/site-packages (from -r src/requirements.txt (line 10)) (2.2.1)\n",
      "Requirement already satisfied: click<9,>=7.1.2 in /opt/conda/lib/python3.7/site-packages (from kfp==2.0.0b12->-r src/requirements.txt (line 1)) (8.1.3)\n",
      "Requirement already satisfied: docstring-parser<1,>=0.7.3 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp==2.0.0b12->-r src/requirements.txt (line 1)) (0.15)\n",
      "Collecting kfp-pipeline-spec==0.2.0\n",
      "  Using cached kfp_pipeline_spec-0.2.0-py3-none-any.whl (12 kB)\n",
      "Requirement already satisfied: kfp-server-api==2.0.0a6 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp==2.0.0b12->-r src/requirements.txt (line 1)) (2.0.0a6)\n",
      "Requirement already satisfied: kubernetes<24,>=8.0.0 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp==2.0.0b12->-r src/requirements.txt (line 1)) (18.20.0)\n",
      "Requirement already satisfied: protobuf<4,>=3.13.0 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp==2.0.0b12->-r src/requirements.txt (line 1)) (3.19.6)\n",
      "Requirement already satisfied: PyYAML<7,>=5.3 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp==2.0.0b12->-r src/requirements.txt (line 1)) (5.4.1)\n",
      "Requirement already satisfied: requests-toolbelt<1,>=0.8.0 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp==2.0.0b12->-r src/requirements.txt (line 1)) (0.10.1)\n",
      "Requirement already satisfied: tabulate<1,>=0.8.6 in /home/jupyter/.local/lib/python3.7/site-packages (from kfp==2.0.0b12->-r src/requirements.txt (line 1)) (0.9.0)\n",
      "Requirement already satisfied: typing-extensions<5,>=3.7.4 in /opt/conda/lib/python3.7/site-packages (from kfp==2.0.0b12->-r src/requirements.txt (line 1)) (4.4.0)\n",
      "Requirement already satisfied: importlib-metadata>=0.12 in /opt/conda/lib/python3.7/site-packages (from pytest==7.2.0->-r src/requirements.txt (line 2)) (6.0.0)\n",
      "Requirement already satisfied: packaging in /home/jupyter/.local/lib/python3.7/site-packages (from pytest==7.2.0->-r src/requirements.txt (line 2)) (21.3)\n",
      "Requirement already satisfied: iniconfig in /home/jupyter/.local/lib/python3.7/site-packages (from pytest==7.2.0->-r src/requirements.txt (line 2)) (2.0.0)\n",
      "Requirement already satisfied: exceptiongroup>=1.0.0rc8 in /home/jupyter/.local/lib/python3.7/site-packages (from pytest==7.2.0->-r src/requirements.txt (line 2)) (1.1.0)\n",
      "Requirement already satisfied: tomli>=1.0.0 in /opt/conda/lib/python3.7/site-packages (from pytest==7.2.0->-r src/requirements.txt (line 2)) (2.0.1)\n",
      "Requirement already satisfied: attrs>=19.2.0 in /opt/conda/lib/python3.7/site-packages (from pytest==7.2.0->-r src/requirements.txt (line 2)) (22.2.0)\n",
      "Requirement already satisfied: pluggy<2.0,>=0.12 in /home/jupyter/.local/lib/python3.7/site-packages (from pytest==7.2.0->-r src/requirements.txt (line 2)) (1.0.0)\n",
      "Requirement already satisfied: proto-plus<2.0.0dev,>=1.22.0 in /opt/conda/lib/python3.7/site-packages (from google-cloud-aiplatform==1.20.0->-r src/requirements.txt (line 4)) (1.22.2)\n",
      "Requirement already satisfied: requests<3.0.0dev,>=2.18.0 in /opt/conda/lib/python3.7/site-packages (from google-api-core==2.10.2->-r src/requirements.txt (line 5)) (2.28.2)\n",
      "Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.56.2 in /opt/conda/lib/python3.7/site-packages (from google-api-core==2.10.2->-r src/requirements.txt (line 5)) (1.58.0)\n",
      "Requirement already satisfied: rsa<5,>=3.1.4 in /opt/conda/lib/python3.7/site-packages (from google-auth==1.35.0->-r src/requirements.txt (line 6)) (4.9)\n",
      "Requirement already satisfied: six>=1.9.0 in /opt/conda/lib/python3.7/site-packages (from google-auth==1.35.0->-r src/requirements.txt (line 6)) (1.16.0)\n",
      "Requirement already satisfied: setuptools>=40.3.0 in /opt/conda/lib/python3.7/site-packages (from google-auth==1.35.0->-r src/requirements.txt (line 6)) (66.1.1)\n",
      "Requirement already satisfied: pyasn1-modules>=0.2.1 in /opt/conda/lib/python3.7/site-packages (from google-auth==1.35.0->-r src/requirements.txt (line 6)) (0.2.8)\n",
      "Requirement already satisfied: cachetools<5.0,>=2.0.0 in /home/jupyter/.local/lib/python3.7/site-packages (from google-auth==1.35.0->-r src/requirements.txt (line 6)) (4.2.4)\n",
      "Requirement already satisfied: google-resumable-media>=0.3.1 in /opt/conda/lib/python3.7/site-packages (from google-cloud-bigquery==1.20.0->-r src/requirements.txt (line 7)) (2.4.1)\n",
      "Requirement already satisfied: grpc-google-iam-v1<1.0.0dev,>=0.12.4 in /opt/conda/lib/python3.7/site-packages (from google-cloud-resource-manager==1.6.3->-r src/requirements.txt (line 9)) (0.12.6)\n",
      "Requirement already satisfied: certifi in /opt/conda/lib/python3.7/site-packages (from kfp-server-api==2.0.0a6->kfp==2.0.0b12->-r src/requirements.txt (line 1)) (2022.12.7)\n",
      "Requirement already satisfied: urllib3>=1.15 in /opt/conda/lib/python3.7/site-packages (from kfp-server-api==2.0.0a6->kfp==2.0.0b12->-r src/requirements.txt (line 1)) (1.26.14)\n",
      "Requirement already satisfied: python-dateutil in /opt/conda/lib/python3.7/site-packages (from kfp-server-api==2.0.0a6->kfp==2.0.0b12->-r src/requirements.txt (line 1)) (2.8.2)\n",
      "Requirement already satisfied: grpcio-status<2.0dev,>=1.33.2 in /opt/conda/lib/python3.7/site-packages (from google-api-core==2.10.2->-r src/requirements.txt (line 5)) (1.48.2)\n",
      "Requirement already satisfied: grpcio<2.0dev,>=1.33.2 in /opt/conda/lib/python3.7/site-packages (from google-api-core==2.10.2->-r src/requirements.txt (line 5)) (1.51.1)\n",
      "Requirement already satisfied: google-crc32c<2.0dev,>=1.0 in /opt/conda/lib/python3.7/site-packages (from google-resumable-media>=0.3.1->google-cloud-bigquery==1.20.0->-r src/requirements.txt (line 7)) (1.5.0)\n",
      "Requirement already satisfied: zipp>=0.5 in /opt/conda/lib/python3.7/site-packages (from importlib-metadata>=0.12->pytest==7.2.0->-r src/requirements.txt (line 2)) (3.12.0)\n",
      "Requirement already satisfied: requests-oauthlib in /opt/conda/lib/python3.7/site-packages (from kubernetes<24,>=8.0.0->kfp==2.0.0b12->-r src/requirements.txt (line 1)) (1.3.1)\n",
      "Requirement already satisfied: websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0 in /opt/conda/lib/python3.7/site-packages (from kubernetes<24,>=8.0.0->kfp==2.0.0b12->-r src/requirements.txt (line 1)) (1.5.0)\n",
      "Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /opt/conda/lib/python3.7/site-packages (from packaging->pytest==7.2.0->-r src/requirements.txt (line 2)) (3.0.9)\n",
      "Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /opt/conda/lib/python3.7/site-packages (from pyasn1-modules>=0.2.1->google-auth==1.35.0->-r src/requirements.txt (line 6)) (0.4.8)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core==2.10.2->-r src/requirements.txt (line 5)) (2.1.1)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /opt/conda/lib/python3.7/site-packages (from requests<3.0.0dev,>=2.18.0->google-api-core==2.10.2->-r src/requirements.txt (line 5)) (3.4)\n",
      "Requirement already satisfied: oauthlib>=3.0.0 in /opt/conda/lib/python3.7/site-packages (from requests-oauthlib->kubernetes<24,>=8.0.0->kfp==2.0.0b12->-r src/requirements.txt (line 1)) (3.2.2)\n",
      "Building wheels for collected packages: kfp\n",
      "  Building wheel for kfp (setup.py) ... \u001b[?25ldone\n",
      "\u001b[?25h  Created wheel for kfp: filename=kfp-2.0.0b12-py3-none-any.whl size=556211 sha256=c317b0a5e39db3b4f728bf16d3bc4982d94324cfff00498765996a91428279ed\n",
      "  Stored in directory: /home/jupyter/.cache/pip/wheels/6a/32/7b/59029a83a3d4addae5589319683a03d963fe61ada3ec2afb96\n",
      "Successfully built kfp\n",
      "Installing collected packages: kfp-pipeline-spec, kfp\n",
      "  Attempting uninstall: kfp-pipeline-spec\n",
      "    Found existing installation: kfp-pipeline-spec 0.1.16\n",
      "    Uninstalling kfp-pipeline-spec-0.1.16:\n",
      "      Successfully uninstalled kfp-pipeline-spec-0.1.16\n",
      "  Attempting uninstall: kfp\n",
      "    Found existing installation: kfp 2.0.0b9\n",
      "    Uninstalling kfp-2.0.0b9:\n",
      "      Successfully uninstalled kfp-2.0.0b9\n",
      "\u001b[33m  WARNING: The scripts dsl-compile, dsl-compile-deprecated and kfp are installed in '/home/jupyter/.local/bin' which is not on PATH.\n",
      "  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\u001b[0m\u001b[33m\n",
      "\u001b[0mSuccessfully installed kfp-2.0.0b12 kfp-pipeline-spec-0.2.0\n"
     ]
    }
   ],
   "source": [
    "!pip install -r src/requirements.txt --user"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f15aada8-e660-4ca5-98d9-56dbb6d33d99",
   "metadata": {},
   "source": [
    "### Setup environment variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "1dca0511",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "if not os.getenv(\"IS_TESTING\"):\n",
    "    # Automatically restart kernel after installs\n",
    "    import IPython\n",
    "\n",
    "    app = IPython.Application.instance()\n",
    "    app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "id": "2cf90436",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Project ID: vertex-ai-test-365213\n",
      "Project Num 370018035372\n"
     ]
    }
   ],
   "source": [
    "PROJECT_ID = \"[your-project-id]\"  # @param {type:\"string\"}\n",
    "     \n",
    "if PROJECT_ID == \"\" or PROJECT_ID is None or PROJECT_ID == \"[your-project-id]\":\n",
    "    # Get your GCP project id from gcloud\n",
    "    shell_output = ! gcloud config list --format 'value(core.project)' 2>/dev/null\n",
    "    PROJECT_ID = shell_output[0]\n",
    "    print(\"Project ID:\", PROJECT_ID)\n",
    "    \n",
    "    shell_output = !  ! gcloud projects list --filter=\"$PROJECT_ID\" --format=\"value(PROJECT_NUMBER)\" 2>/dev/null\n",
    "    PROJECT_NUM = shell_output[0]\n",
    "    print(\"Project Num\", PROJECT_NUM)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "18ddbb36",
   "metadata": {},
   "outputs": [],
   "source": [
    "REGION = \"[your-region]\"  # @param {type: \"string\"}\n",
    "\n",
    "if REGION == \"[your-region]\":\n",
    "    REGION = \"us-central1\"\n",
    "    \n",
    "BUCKET_NAME = f\"mlops-2-{PROJECT_ID}\"\n",
    "EXPERIMENT_NAME = \"mlops-2-experiment\"\n",
    "PIPELINE_NAME = \"mlops-2-pipeline\"\n",
    "ENDPOINT_NAME = \"mlops-2-endpoint\"\n",
    "REPOSITORY_NAME = f\"mlops-2-{PROJECT_ID}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "8c24e1b0",
   "metadata": {
    "collapsed": true,
    "jupyter": {
     "outputs_hidden": true
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating gs://mlops-coaching-2-vertex-ai-test-365213/...\n",
      "ServiceException: 409 A Cloud Storage bucket named 'mlops-coaching-2-vertex-ai-test-365213' already exists. Try another name. Bucket names must be globally unique across all Google Cloud projects, including those outside of your organization.\n"
     ]
    }
   ],
   "source": [
    "!gsutil mb -c regional -l $REGION gs://$BUCKET_NAME"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "57a66b6f",
   "metadata": {},
   "source": [
    "## Automated MLOps pipeline creation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a84feaa",
   "metadata": {},
   "source": [
    "### Create a script containing your Vertex AI/Kubeflow Pipeline to compile the pipeline into `pipeline.yaml`\n",
    "\n",
    "> <font color='green'>**Task 1**</font>\n",
    ">\n",
    "> Create a Python script `src/pipeline.py` that creates a file name `pipeline.yaml` from the Kubeflow pipeline you developed last week. The output file should be in YAML and not JSON format.\n",
    ">\n",
    "> If you were unable to produce a Kubeflow pipeline last week, please use the one provided below. Otherwise, replace it with your own."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "76d71ce2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting src/pipeline.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile src/pipeline.py\n",
    "\n",
    "#### Optionally, you can customise the pipeline you are deploying by inserting the pipeline from part 1 ####\n",
    "\n",
    "\n",
    "from typing import NamedTuple\n",
    "\n",
    "from kfp.dsl import pipeline\n",
    "from kfp.dsl import component\n",
    "from kfp import compiler\n",
    "\n",
    "@component() \n",
    "def concat(a: str, b: str) -> str:\n",
    "    return a + b\n",
    "\n",
    "@component\n",
    "def reverse(a: str) -> NamedTuple(\"outputs\", [(\"before\", str), (\"after\", str)]):\n",
    "    return a, a[::-1]\n",
    "\n",
    "@pipeline(name=\"mlops-workshop-pipeline\")\n",
    "def basic_pipeline(a: str='stres', b: str='sed'):\n",
    "    concat_task = concat(a=a, b=b)\n",
    "    reverse_task = reverse(a=concat_task.output)\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    compiler.Compiler().compile(pipeline_func=basic_pipeline, package_path=\"pipeline.yaml\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fe2d4148",
   "metadata": {},
   "source": [
    "Using the next command, you can test the materialized pipeline generated by your script. You can view the output in a file named `pipeline.yaml`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "54c94289",
   "metadata": {},
   "outputs": [],
   "source": [
    "!python src/pipeline.py"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "66bb273c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "# PIPELINE DEFINITION\n",
      "# Name: mlops-coaching-pipeline\n",
      "# Inputs:\n",
      "#    a: str [Default: 'stres']\n",
      "#    b: str [Default: 'sed']\n",
      "components:\n",
      "  comp-concat:\n",
      "    executorLabel: exec-concat\n",
      "    inputDefinitions:\n",
      "      parameters:\n",
      "        a:\n",
      "          parameterType: STRING\n",
      "        b:\n",
      "          parameterType: STRING\n",
      "    outputDefinitions:\n",
      "      parameters:\n",
      "        Output:\n",
      "          parameterType: STRING\n",
      "  comp-reverse:\n",
      "    executorLabel: exec-reverse\n"
     ]
    }
   ],
   "source": [
    "!head -n20 pipeline.yaml"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ca25b67-c432-4d42-8a10-2b9b94e7a980",
   "metadata": {},
   "source": [
    "### Test the Pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3ef73fcf-6a9c-49da-bb65-7f724a412c5e",
   "metadata": {},
   "source": [
    "> <font color='green'>**Task 2**</font>\n",
    "> Write unit/integration tests for the pipeline you created to ensure the component logic that you added works as expected"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "88e162c6-1cbb-456c-8b5c-d513fbe34064",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p src/tests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8186aaf9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing src/tests/test_pipeline.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile src/tests/test_pipeline.py\n",
    "\n",
    "import unittest\n",
    "from pipeline import concat, reverse, basic_pipeline\n",
    "\n",
    "class TestBasicPipeline(unittest.TestCase):\n",
    "    # def setUp(self):\n",
    "        # Get relevant component\n",
    "    \n",
    "    def test_concat_component(self):\n",
    "        self.assertEqual(concat.python_func(3, 3), 6)\n",
    "\n",
    "    def test_reverse(self):\n",
    "        self.assertEqual(reverse.python_func(\"stressed\")[1], \"desserts\")\n",
    "\n",
    "    def test_pipeline(self):\n",
    "        pass\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    unittest.main()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "695a7b41-9996-42c1-8849-70aef63273f9",
   "metadata": {},
   "source": [
    "Using the next command, you can run the tests in the script using python `unittest` test runner. It discovers all the test files that start with `test_*`\n",
    "\n",
    "You can also use other testing framework of your choice (e.g. `pytest`)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "191cfe48-0206-477c-a894-7c4ca66d6eda",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "...\n",
      "----------------------------------------------------------------------\n",
      "Ran 3 tests in 0.000s\n",
      "\n",
      "OK\n"
     ]
    }
   ],
   "source": [
    "!PYTHONPATH=src python -m unittest discover -s src/tests/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a576513",
   "metadata": {},
   "source": [
    "### Create a script to submit your compile kubeflow pipeline (`pipeline.yaml`) to Vertex AI"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "20c708bb",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting src/submit-pipeline.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile src/submit-pipeline.py\n",
    "import os\n",
    "\n",
    "from google.cloud import aiplatform\n",
    "import google.auth\n",
    "\n",
    "PROJECT_ID = os.getenv(\"PROJECT_ID\")\n",
    "if not PROJECT_ID:\n",
    "    creds, PROJECT_ID = google.auth.default()\n",
    "\n",
    "REGION = os.environ[\"REGION\"]\n",
    "BUCKET_NAME = os.environ[\"BUCKET_NAME\"]\n",
    "EXPERIMENT_NAME = os.environ[\"EXPERIMENT_NAME\"]\n",
    "ENDPOINT_NAME = os.environ[\"ENDPOINT_NAME\"]\n",
    "PIPELINE_NAME = os.environ[\"PIPELINE_NAME\"]\n",
    "ENABLE_CACHING = os.getenv(\"CACHE_PIPELINE\", 'true').lower() in ('true', '1', 't')\n",
    "\n",
    "aiplatform.init(project=PROJECT_ID, location=REGION)\n",
    "sync_pipeline = os.getenv(\"SUBMIT_PIPELINE_SYNC\", 'False').lower() in ('true', '1', 't')\n",
    "\n",
    "job = aiplatform.PipelineJob(\n",
    "    display_name=PIPELINE_NAME,\n",
    "    template_path='pipeline.yaml',\n",
    "    location=REGION,\n",
    "    project=PROJECT_ID,\n",
    "    enable_caching=ENABLE_CACHING,\n",
    "    pipeline_root=f'gs://{BUCKET_NAME}'\n",
    ")\n",
    "print(f\"Submitting pipeline {PIPELINE_NAME} in experiment {EXPERIMENT_NAME}.\")\n",
    "job.submit(experiment=EXPERIMENT_NAME)\n",
    "\n",
    "if sync_pipeline:\n",
    "    job.wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a3de2c1",
   "metadata": {},
   "source": [
    "Let's test this script in the Notebook. You can check the pipeline's status by clicking on the link printed by the script."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "7d630863",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: REGION=us-central1\n",
      "env: BUCKET_NAME=mlops-coaching-2-vertex-ai-test-365213\n",
      "env: EXPERIMENT_NAME=mlops-coaching-2-experiment\n",
      "env: PIPELINE_NAME=mlops-coaching-2-pipeline\n",
      "env: ENDPOINT_NAME=mlops-coaching-2-endpoint\n",
      "env: SUBMIT_PIPELINE_SYNC=1\n",
      "Submitting pipeline mlops-coaching-2-pipeline in experiment mlops-coaching-2-experiment.\n",
      "Creating PipelineJob\n",
      "PipelineJob created. Resource name: projects/370018035372/locations/us-central1/pipelineJobs/mlops-coaching-pipeline-20230210104322\n",
      "To use this PipelineJob in another session:\n",
      "pipeline_job = aiplatform.PipelineJob.get('projects/370018035372/locations/us-central1/pipelineJobs/mlops-coaching-pipeline-20230210104322')\n",
      "View Pipeline Job:\n",
      "https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/mlops-coaching-pipeline-20230210104322?project=370018035372\n",
      "Associating projects/370018035372/locations/us-central1/pipelineJobs/mlops-coaching-pipeline-20230210104322 to Experiment: mlops-coaching-2-experiment\n",
      "PipelineJob projects/370018035372/locations/us-central1/pipelineJobs/mlops-coaching-pipeline-20230210104322 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/370018035372/locations/us-central1/pipelineJobs/mlops-coaching-pipeline-20230210104322 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/370018035372/locations/us-central1/pipelineJobs/mlops-coaching-pipeline-20230210104322 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob projects/370018035372/locations/us-central1/pipelineJobs/mlops-coaching-pipeline-20230210104322 current state:\n",
      "PipelineState.PIPELINE_STATE_RUNNING\n",
      "PipelineJob run completed. Resource name: projects/370018035372/locations/us-central1/pipelineJobs/mlops-coaching-pipeline-20230210104322\n"
     ]
    }
   ],
   "source": [
    "%set_env REGION=$REGION\n",
    "%set_env BUCKET_NAME=$BUCKET_NAME\n",
    "%set_env EXPERIMENT_NAME=$EXPERIMENT_NAME\n",
    "%set_env PIPELINE_NAME=$PIPELINE_NAME\n",
    "%set_env ENDPOINT_NAME=$ENDPOINT_NAME\n",
    "%set_env SUBMIT_PIPELINE_SYNC=1\n",
    "\n",
    "!python src/submit-pipeline.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6faaf9dd",
   "metadata": {},
   "source": [
    "### Create a pipeline template in Artifact Registry from `pipeline.yaml`\n",
    "\n",
    "A pipeline template is a resource that you can use to publish a workflow definition so that it can be reused multiple times, by a single user or by multiple users. This feature is [documented here](https://cloud.google.com/vertex-ai/docs/pipelines/create-pipeline-template).\n",
    "\n",
    "The Kubeflow Pipelines SDK registry client is a new client interface that you can use with a compatible registry server, such as Artifact Registry, for version control of your Kubeflow Pipelines (KFP) templates.\n",
    "\n",
    "> <font color='green'>**Task 3**</font>\n",
    ">\n",
    "> Create a Python script `src/create-pipeline-template.py` that uploads `pipeline.yaml` to the Vertex AI Pipeline registry.\n",
    ">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba4e22a1",
   "metadata": {},
   "outputs": [],
   "source": [
    "!gcloud artifacts repositories create mlops2-repo --location=$REGION --repository-format=KFP"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "bc5a84da",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing src/create-pipeline-template.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile src/create-pipeline-template.py\n",
    "import os\n",
    "import google.auth\n",
    "\n",
    "from kfp.registry import RegistryClient\n",
    "\n",
    "PROJECT_ID = os.getenv(\"PROJECT_ID\")\n",
    "if not PROJECT_ID:\n",
    "    creds, PROJECT_ID = google.auth.default()\n",
    "REGION = os.environ[\"REGION\"]\n",
    "\n",
    "## Your code goes below this line\n",
    "\n",
    "client = RegistryClient(host=f\"https://{REGION}-kfp.pkg.dev/{PROJECT_ID}/mlops2-repo\")\n",
    "\n",
    "template_name, template_version = client.upload_pipeline(\n",
    "  file_name=\"pipeline.yaml\",\n",
    "  tags=[\"v1\", \"latest\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "29c7182b",
   "metadata": {},
   "outputs": [],
   "source": [
    "!python src/create-pipeline-template.py"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a82cf0be",
   "metadata": {},
   "source": [
    "### Automate Kubeflow pipeline compilation, template generation, and execution through Cloud Build\n",
    "\n",
    "Cloud Build is a service that executes your builds on Google Cloud. In this exercise, we want to use it to both compile and run your machine learning pipeline. For more information, please refer to the [Cloud Build documentation](https://cloud.google.com/build/docs/overview)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "243972cd",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting src/cloudbuild.yaml\n"
     ]
    }
   ],
   "source": [
    "%%writefile src/cloudbuild.yaml\n",
    "steps:\n",
    "  # Install dependencies\n",
    "  - name: 'python'\n",
    "    entrypoint: 'pip'\n",
    "    args: [\"install\", \"-r\", \"requirements.txt\", \"--user\"]\n",
    "\n",
    "  # Compile pipeline\n",
    "  - name: 'python'\n",
    "    entrypoint: 'python'\n",
    "    args: ['pipeline.py']\n",
    "    id: 'compile'\n",
    "\n",
    "  # Test the Pipeline Components \n",
    "  - name: 'python'\n",
    "    entrypoint: 'python'\n",
    "    args: ['-m', 'unittest', 'discover', 'tests/']\n",
    "    id: 'test_pipeline'\n",
    "    waitFor: ['compile']\n",
    "\n",
    "  # Upload compiled pipeline to GCS\n",
    "  - name: 'gcr.io/cloud-builders/gsutil'\n",
    "    args: ['cp', 'pipeline.yaml', 'gs://${_BUCKET_NAME}']\n",
    "    id: 'upload'\n",
    "    waitFor: ['test_pipeline']\n",
    "        \n",
    "  # Run the Vertex AI Pipeline (synchronously for test/qa environment) with caching enabled.\n",
    "  - name: 'python'\n",
    "    id: 'test'\n",
    "    entrypoint: 'python'\n",
    "    env: ['BUCKET_NAME=${_BUCKET_NAME}', 'EXPERIMENT_NAME=qa-${_EXPERIMENT_NAME}', 'PIPELINE_NAME=${_PIPELINE_NAME}',\n",
    "          'REGION=${_REGION}', 'ENDPOINT_NAME=qa-${_ENDPOINT_NAME}', 'SUBMIT_PIPELINE_SYNC=true', 'ENABLE_CACHING=true']\n",
    "    args: ['submit-pipeline.py']\n",
    "\n",
    "  # Create pipeline template and upload it to the artifact registry\n",
    "  - name: 'python'\n",
    "    id: 'template'\n",
    "    entrypoint: 'python'\n",
    "    env: ['REGION=${_REGION}']\n",
    "    args: ['create-pipeline-template.py']\n",
    "    \n",
    "  # Run the Vertex AI Pipeline (asynchronously for prod environment) with caching disabled.\n",
    "  # In a real production scenario, this would run in a different GCP project.\n",
    "  - name: 'python'\n",
    "    id: 'prod'\n",
    "    entrypoint: 'python'\n",
    "    env: ['BUCKET_NAME=${_BUCKET_NAME}', 'EXPERIMENT_NAME=prod-${_EXPERIMENT_NAME}', 'PIPELINE_NAME=${_PIPELINE_NAME}',\n",
    "          'REGION=${_REGION}', 'ENDPOINT_NAME=prod-${_ENDPOINT_NAME}', 'SUBMIT_PIPELINE_SYNC=false', 'ENABLE_CACHING=false']\n",
    "    args: ['submit-pipeline.py']"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4958d76",
   "metadata": {},
   "source": [
    "Cloud Build uses a special service account to execute builds on your behalf. When you enable the Cloud Build API on a Google Cloud project, the Cloud Build service account is automatically created and granted the Cloud Build Service Account role for the project. This role gives the service account permissions to perform several tasks, however you can grant more permissions to the service account to perform additional tasks. [This page](https://cloud.google.com/build/docs/securing-builds/configure-access-for-cloud-build-service-account) explains how to grant and revoke permissions to the Cloud Build service account.\n",
    "\n",
    "For Cloud Build to be able to deploy your pipeline, you need to give its' service account `{PROJECT_NUMBER}@cloudbuild.gserviceaccount.com` the **Vertex AI User** and **Service Account User** role. Note that it may take up to 5 minutes until the new permissions propagate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "d6d3ea3b",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating temporary tarball archive of 8 file(s) totalling 6.5 KiB before compression.\n",
      "Uploading tarball of [./src] to [gs://vertex-ai-test-365213_cloudbuild/source/1676026036.796529-80b198a42908461cac0e8895c5563b37.tgz]\n",
      "Created [https://cloudbuild.googleapis.com/v1/projects/vertex-ai-test-365213/locations/global/builds/599caf7a-a504-481d-bfb9-050191d41ee2].\n",
      "Logs are available at [ https://console.cloud.google.com/cloud-build/builds/599caf7a-a504-481d-bfb9-050191d41ee2?project=370018035372 ].\n",
      "----------------------------- REMOTE BUILD OUTPUT ------------------------------\n",
      "starting build \"599caf7a-a504-481d-bfb9-050191d41ee2\"\n",
      "\n",
      "FETCHSOURCE\n",
      "Fetching storage object: gs://vertex-ai-test-365213_cloudbuild/source/1676026036.796529-80b198a42908461cac0e8895c5563b37.tgz#1676026037036540\n",
      "Copying gs://vertex-ai-test-365213_cloudbuild/source/1676026036.796529-80b198a42908461cac0e8895c5563b37.tgz#1676026037036540...\n",
      "/ [1 files][  3.3 KiB/  3.3 KiB]                                                \n",
      "Operation completed over 1 objects/3.3 KiB.\n",
      "BUILD\n",
      "Starting Step #0\n",
      "Step #0: Pulling image: python\n",
      "Step #0: Using default tag: latest\n",
      "Step #0: latest: Pulling from library/python\n",
      "Step #0: 699c8a97647f: Pulling fs layer\n",
      "Step #0: 86cd158b89fd: Pulling fs layer\n",
      "Step #0: a226e961cfaa: Pulling fs layer\n",
      "Step #0: 4cec535da207: Pulling fs layer\n",
      "Step #0: 225fdd30e1a3: Pulling fs layer\n",
      "Step #0: 356a16c6c201: Pulling fs layer\n",
      "Step #0: 839c36583c9f: Pulling fs layer\n",
      "Step #0: 366ea971a2dc: Pulling fs layer\n",
      "Step #0: e4b89ee4cc7d: Pulling fs layer\n",
      "Step #0: 356a16c6c201: Waiting\n",
      "Step #0: 839c36583c9f: Waiting\n",
      "Step #0: 366ea971a2dc: Waiting\n",
      "Step #0: e4b89ee4cc7d: Waiting\n",
      "Step #0: 4cec535da207: Waiting\n",
      "Step #0: 225fdd30e1a3: Waiting\n",
      "Step #0: 86cd158b89fd: Verifying Checksum\n",
      "Step #0: 86cd158b89fd: Download complete\n",
      "Step #0: a226e961cfaa: Verifying Checksum\n",
      "Step #0: a226e961cfaa: Download complete\n",
      "Step #0: 699c8a97647f: Verifying Checksum\n",
      "Step #0: 699c8a97647f: Download complete\n",
      "Step #0: 356a16c6c201: Verifying Checksum\n",
      "Step #0: 356a16c6c201: Download complete\n",
      "Step #0: 4cec535da207: Verifying Checksum\n",
      "Step #0: 4cec535da207: Download complete\n",
      "Step #0: 366ea971a2dc: Verifying Checksum\n",
      "Step #0: 366ea971a2dc: Download complete\n",
      "Step #0: 839c36583c9f: Download complete\n",
      "Step #0: e4b89ee4cc7d: Verifying Checksum\n",
      "Step #0: e4b89ee4cc7d: Download complete\n",
      "Step #0: 225fdd30e1a3: Verifying Checksum\n",
      "Step #0: 225fdd30e1a3: Download complete\n",
      "Step #0: 699c8a97647f: Pull complete\n",
      "Step #0: 86cd158b89fd: Pull complete\n",
      "Step #0: a226e961cfaa: Pull complete\n",
      "Step #0: 4cec535da207: Pull complete\n",
      "Step #0: 225fdd30e1a3: Pull complete\n",
      "Step #0: 356a16c6c201: Pull complete\n",
      "Step #0: 839c36583c9f: Pull complete\n",
      "Step #0: 366ea971a2dc: Pull complete\n",
      "Step #0: e4b89ee4cc7d: Pull complete\n",
      "Step #0: Digest: sha256:91778894494195bc996601622cfc2e545852061c0b756ef4bafe87506fadcc61\n",
      "Step #0: Status: Downloaded newer image for python:latest\n",
      "Step #0: docker.io/library/python:latest\n",
      "Step #0: Collecting kfp==2.0.0b12\n",
      "Step #0:   Downloading kfp-2.0.0-beta.12.tar.gz (492 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 492.9/492.9 kB 31.9 MB/s eta 0:00:00\n",
      "Step #0:   Preparing metadata (setup.py): started\n",
      "Step #0:   Preparing metadata (setup.py): finished with status 'done'\n",
      "Step #0: Collecting pytest==7.2.0\n",
      "Step #0:   Downloading pytest-7.2.0-py3-none-any.whl (316 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 316.8/316.8 kB 36.2 MB/s eta 0:00:00\n",
      "Step #0: Collecting pytz==2022.7\n",
      "Step #0:   Downloading pytz-2022.7-py2.py3-none-any.whl (499 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 499.4/499.4 kB 48.7 MB/s eta 0:00:00\n",
      "Step #0: Collecting google-cloud-aiplatform==1.20.0\n",
      "Step #0:   Downloading google_cloud_aiplatform-1.20.0-py2.py3-none-any.whl (2.3 MB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.3/2.3 MB 85.2 MB/s eta 0:00:00\n",
      "Step #0: Collecting google-api-core==2.10.2\n",
      "Step #0:   Downloading google_api_core-2.10.2-py3-none-any.whl (115 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 115.6/115.6 kB 19.0 MB/s eta 0:00:00\n",
      "Step #0: Collecting google-auth==1.35.0\n",
      "Step #0:   Downloading google_auth-1.35.0-py2.py3-none-any.whl (152 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 152.9/152.9 kB 25.4 MB/s eta 0:00:00\n",
      "Step #0: Collecting google-cloud-bigquery==1.20.0\n",
      "Step #0:   Downloading google_cloud_bigquery-1.20.0-py2.py3-none-any.whl (154 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 155.0/155.0 kB 24.5 MB/s eta 0:00:00\n",
      "Step #0: Collecting google-cloud-core==1.7.3\n",
      "Step #0:   Downloading google_cloud_core-1.7.3-py2.py3-none-any.whl (28 kB)\n",
      "Step #0: Collecting google-cloud-resource-manager==1.6.3\n",
      "Step #0:   Downloading google_cloud_resource_manager-1.6.3-py2.py3-none-any.whl (233 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 233.8/233.8 kB 35.6 MB/s eta 0:00:00\n",
      "Step #0: Collecting google-cloud-storage==2.2.1\n",
      "Step #0:   Downloading google_cloud_storage-2.2.1-py2.py3-none-any.whl (107 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 107.1/107.1 kB 19.8 MB/s eta 0:00:00\n",
      "Step #0: Collecting click<9,>=7.1.2\n",
      "Step #0:   Downloading click-8.1.3-py3-none-any.whl (96 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 96.6/96.6 kB 17.5 MB/s eta 0:00:00\n",
      "Step #0: Collecting docstring-parser<1,>=0.7.3\n",
      "Step #0:   Downloading docstring_parser-0.15-py3-none-any.whl (36 kB)\n",
      "Step #0: Collecting kfp-pipeline-spec==0.2.0\n",
      "Step #0:   Downloading kfp_pipeline_spec-0.2.0-py3-none-any.whl (12 kB)\n",
      "Step #0: Collecting kfp-server-api==2.0.0a6\n",
      "Step #0:   Downloading kfp-server-api-2.0.0a6.tar.gz (60 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.5/60.5 kB 10.6 MB/s eta 0:00:00\n",
      "Step #0:   Preparing metadata (setup.py): started\n",
      "Step #0:   Preparing metadata (setup.py): finished with status 'done'\n",
      "Step #0: Collecting kubernetes<24,>=8.0.0\n",
      "Step #0:   Downloading kubernetes-23.6.0-py2.py3-none-any.whl (1.5 MB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.5/1.5 MB 69.1 MB/s eta 0:00:00\n",
      "Step #0: Collecting protobuf<4,>=3.13.0\n",
      "Step #0:   Downloading protobuf-3.20.3-py2.py3-none-any.whl (162 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 162.1/162.1 kB 26.2 MB/s eta 0:00:00\n",
      "Step #0: Collecting PyYAML<7,>=5.3\n",
      "Step #0:   Downloading PyYAML-6.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (757 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 757.9/757.9 kB 66.3 MB/s eta 0:00:00\n",
      "Step #0: Collecting requests-toolbelt<1,>=0.8.0\n",
      "Step #0:   Downloading requests_toolbelt-0.10.1-py2.py3-none-any.whl (54 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 54.5/54.5 kB 10.6 MB/s eta 0:00:00\n",
      "Step #0: Collecting tabulate<1,>=0.8.6\n",
      "Step #0:   Downloading tabulate-0.9.0-py3-none-any.whl (35 kB)\n",
      "Step #0: Collecting attrs>=19.2.0\n",
      "Step #0:   Downloading attrs-22.2.0-py3-none-any.whl (60 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.0/60.0 kB 11.4 MB/s eta 0:00:00\n",
      "Step #0: Collecting iniconfig\n",
      "Step #0:   Downloading iniconfig-2.0.0-py3-none-any.whl (5.9 kB)\n",
      "Step #0: Collecting packaging\n",
      "Step #0:   Downloading packaging-23.0-py3-none-any.whl (42 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 42.7/42.7 kB 8.0 MB/s eta 0:00:00\n",
      "Step #0: Collecting pluggy<2.0,>=0.12\n",
      "Step #0:   Downloading pluggy-1.0.0-py2.py3-none-any.whl (13 kB)\n",
      "Step #0: Collecting google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.*,!=2.4.*,!=2.5.*,!=2.6.*,!=2.7.*,<3.0.0dev,>=1.32.0\n",
      "Step #0:   Downloading google_api_core-2.11.0-py3-none-any.whl (120 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 120.3/120.3 kB 21.5 MB/s eta 0:00:00\n",
      "Step #0: Collecting proto-plus<2.0.0dev,>=1.22.0\n",
      "Step #0:   Downloading proto_plus-1.22.2-py3-none-any.whl (47 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 47.9/47.9 kB 9.0 MB/s eta 0:00:00\n",
      "Step #0: Collecting packaging\n",
      "Step #0:   Downloading packaging-21.3-py3-none-any.whl (40 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 40.8/40.8 kB 7.3 MB/s eta 0:00:00\n",
      "Step #0: Collecting googleapis-common-protos<2.0dev,>=1.56.2\n",
      "Step #0:   Downloading googleapis_common_protos-1.58.0-py2.py3-none-any.whl (223 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 223.0/223.0 kB 37.1 MB/s eta 0:00:00\n",
      "Step #0: Collecting requests<3.0.0dev,>=2.18.0\n",
      "Step #0:   Downloading requests-2.28.2-py3-none-any.whl (62 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.8/62.8 kB 11.9 MB/s eta 0:00:00\n",
      "Step #0: Collecting cachetools<5.0,>=2.0.0\n",
      "Step #0:   Downloading cachetools-4.2.4-py3-none-any.whl (10 kB)\n",
      "Step #0: Collecting pyasn1-modules>=0.2.1\n",
      "Step #0:   Downloading pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 155.3/155.3 kB 25.3 MB/s eta 0:00:00\n",
      "Step #0: Requirement already satisfied: setuptools>=40.3.0 in /usr/local/lib/python3.11/site-packages (from google-auth==1.35.0->-r requirements.txt (line 6)) (65.5.1)\n",
      "Step #0: Collecting six>=1.9.0\n",
      "Step #0:   Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)\n",
      "Step #0: Collecting rsa<5,>=3.1.4\n",
      "Step #0:   Downloading rsa-4.9-py3-none-any.whl (34 kB)\n",
      "Step #0: Collecting google-resumable-media>=0.3.1\n",
      "Step #0:   Downloading google_resumable_media-2.4.1-py2.py3-none-any.whl (77 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.7/77.7 kB 14.6 MB/s eta 0:00:00\n",
      "Step #0: Collecting grpc-google-iam-v1<1.0.0dev,>=0.12.4\n",
      "Step #0:   Downloading grpc_google_iam_v1-0.12.6-py2.py3-none-any.whl (26 kB)\n",
      "Step #0: Collecting urllib3>=1.15\n",
      "Step #0:   Downloading urllib3-1.26.14-py2.py3-none-any.whl (140 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.6/140.6 kB 24.0 MB/s eta 0:00:00\n",
      "Step #0: Collecting certifi\n",
      "Step #0:   Downloading certifi-2022.12.7-py3-none-any.whl (155 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 155.3/155.3 kB 24.9 MB/s eta 0:00:00\n",
      "Step #0: Collecting python-dateutil\n",
      "Step #0:   Downloading python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 247.7/247.7 kB 38.9 MB/s eta 0:00:00\n",
      "Step #0: Collecting grpcio<2.0dev,>=1.33.2\n",
      "Step #0:   Downloading grpcio-1.51.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.8 MB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.8/4.8 MB 60.7 MB/s eta 0:00:00\n",
      "Step #0: Collecting grpcio-status<2.0dev,>=1.33.2\n",
      "Step #0:   Downloading grpcio_status-1.51.1-py3-none-any.whl (5.1 kB)\n",
      "Step #0: Collecting google-crc32c<2.0dev,>=1.0\n",
      "Step #0:   Downloading google_crc32c-1.5.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (32 kB)\n",
      "Step #0: Collecting websocket-client!=0.40.0,!=0.41.*,!=0.42.*,>=0.32.0\n",
      "Step #0:   Downloading websocket_client-1.5.1-py3-none-any.whl (55 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 55.9/55.9 kB 10.0 MB/s eta 0:00:00\n",
      "Step #0: Collecting requests-oauthlib\n",
      "Step #0:   Downloading requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)\n",
      "Step #0: Collecting pyparsing!=3.0.5,>=2.0.2\n",
      "Step #0:   Downloading pyparsing-3.0.9-py3-none-any.whl (98 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 98.3/98.3 kB 17.8 MB/s eta 0:00:00\n",
      "Step #0: Collecting pyasn1<0.5.0,>=0.4.6\n",
      "Step #0:   Downloading pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 77.1/77.1 kB 13.5 MB/s eta 0:00:00\n",
      "Step #0: Collecting charset-normalizer<4,>=2\n",
      "Step #0:   Downloading charset_normalizer-3.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (196 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 196.8/196.8 kB 32.1 MB/s eta 0:00:00\n",
      "Step #0: Collecting idna<4,>=2.5\n",
      "Step #0:   Downloading idna-3.4-py3-none-any.whl (61 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.5/61.5 kB 10.1 MB/s eta 0:00:00\n",
      "Step #0: Collecting grpcio-status<2.0dev,>=1.33.2\n",
      "Step #0:   Downloading grpcio_status-1.50.0-py3-none-any.whl (14 kB)\n",
      "Step #0:   Downloading grpcio_status-1.49.1-py3-none-any.whl (14 kB)\n",
      "Step #0:   Downloading grpcio_status-1.48.2-py3-none-any.whl (14 kB)\n",
      "Step #0: Collecting oauthlib>=3.0.0\n",
      "Step #0:   Downloading oauthlib-3.2.2-py3-none-any.whl (151 kB)\n",
      "Step #0:      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 151.7/151.7 kB 24.1 MB/s eta 0:00:00\n",
      "Step #0: Building wheels for collected packages: kfp, kfp-server-api\n",
      "Step #0:   Building wheel for kfp (setup.py): started\n",
      "Step #0:   Building wheel for kfp (setup.py): finished with status 'done'\n",
      "Step #0:   Created wheel for kfp: filename=kfp-2.0.0b12-py3-none-any.whl size=556211 sha256=35d46bee13eb4af71e8f40987770d190613c7c65dac63bbf21307f66f0fde06a\n",
      "Step #0:   Stored in directory: /builder/home/.cache/pip/wheels/db/0b/77/ca9f0382ac59bfce6172a40920de6eaa8ccaa698118a51b97a\n",
      "Step #0:   Building wheel for kfp-server-api (setup.py): started\n",
      "Step #0:   Building wheel for kfp-server-api (setup.py): finished with status 'done'\n",
      "Step #0:   Created wheel for kfp-server-api: filename=kfp_server_api-2.0.0a6-py3-none-any.whl size=104199 sha256=8ff6572f079e041d698f5eb6559b10a52ae0c24828eaee9b4cae5cb9671bbcc8\n",
      "Step #0:   Stored in directory: /builder/home/.cache/pip/wheels/73/8b/1e/2ebf3b125b030ca9e5d5d8b56fa51e5cb7b61bdef7dde681de\n",
      "Step #0: Successfully built kfp kfp-server-api\n",
      "Step #0: Installing collected packages: pytz, pyasn1, charset-normalizer, websocket-client, urllib3, tabulate, six, rsa, PyYAML, pyparsing, pyasn1-modules, protobuf, pluggy, oauthlib, iniconfig, idna, grpcio, google-crc32c, docstring-parser, click, certifi, cachetools, attrs, requests, python-dateutil, proto-plus, packaging, kfp-pipeline-spec, googleapis-common-protos, google-resumable-media, google-auth, requests-toolbelt, requests-oauthlib, pytest, kfp-server-api, grpcio-status, google-api-core, kubernetes, grpc-google-iam-v1, google-cloud-core, google-cloud-storage, google-cloud-resource-manager, google-cloud-bigquery, kfp, google-cloud-aiplatform\n",
      "Step #0:   WARNING: The script normalizer is installed in '/builder/home/.local/bin' which is not on PATH.\n",
      "Step #0:   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\n",
      "Step #0:   WARNING: The script wsdump is installed in '/builder/home/.local/bin' which is not on PATH.\n",
      "Step #0:   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\n",
      "Step #0:   WARNING: The script tabulate is installed in '/builder/home/.local/bin' which is not on PATH.\n",
      "Step #0:   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\n",
      "Step #0:   WARNING: The scripts pyrsa-decrypt, pyrsa-encrypt, pyrsa-keygen, pyrsa-priv2pub, pyrsa-sign and pyrsa-verify are installed in '/builder/home/.local/bin' which is not on PATH.\n",
      "Step #0:   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\n",
      "Step #0:   WARNING: The scripts py.test and pytest are installed in '/builder/home/.local/bin' which is not on PATH.\n",
      "Step #0:   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\n",
      "Step #0:   WARNING: The scripts dsl-compile, dsl-compile-deprecated and kfp are installed in '/builder/home/.local/bin' which is not on PATH.\n",
      "Step #0:   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\n",
      "Step #0:   WARNING: The script tb-gcp-uploader is installed in '/builder/home/.local/bin' which is not on PATH.\n",
      "Step #0:   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.\n",
      "Step #0: Successfully installed PyYAML-6.0 attrs-22.2.0 cachetools-4.2.4 certifi-2022.12.7 charset-normalizer-3.0.1 click-8.1.3 docstring-parser-0.15 google-api-core-2.10.2 google-auth-1.35.0 google-cloud-aiplatform-1.20.0 google-cloud-bigquery-1.20.0 google-cloud-core-1.7.3 google-cloud-resource-manager-1.6.3 google-cloud-storage-2.2.1 google-crc32c-1.5.0 google-resumable-media-2.4.1 googleapis-common-protos-1.58.0 grpc-google-iam-v1-0.12.6 grpcio-1.51.1 grpcio-status-1.48.2 idna-3.4 iniconfig-2.0.0 kfp-2.0.0b12 kfp-pipeline-spec-0.2.0 kfp-server-api-2.0.0a6 kubernetes-23.6.0 oauthlib-3.2.2 packaging-21.3 pluggy-1.0.0 proto-plus-1.22.2 protobuf-3.20.3 pyasn1-0.4.8 pyasn1-modules-0.2.8 pyparsing-3.0.9 pytest-7.2.0 python-dateutil-2.8.2 pytz-2022.7 requests-2.28.2 requests-oauthlib-1.3.1 requests-toolbelt-0.10.1 rsa-4.9 six-1.16.0 tabulate-0.9.0 urllib3-1.26.14 websocket-client-1.5.1\n",
      "Step #0: WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv\n",
      "Step #0: \n",
      "Step #0: [notice] A new release of pip available: 22.3.1 -> 23.0\n",
      "Step #0: [notice] To update, run: pip install --upgrade pip\n",
      "Finished Step #0\n",
      "Starting Step #1 - \"compile\"\n",
      "Step #1 - \"compile\": Already have image: python\n",
      "Finished Step #1 - \"compile\"\n",
      "Starting Step #2 - \"test_pipeline\"\n",
      "Step #2 - \"test_pipeline\": Already have image: python\n",
      "Step #2 - \"test_pipeline\": ...\n",
      "Step #2 - \"test_pipeline\": ----------------------------------------------------------------------\n",
      "Step #2 - \"test_pipeline\": Ran 3 tests in 0.001s\n",
      "Step #2 - \"test_pipeline\": \n",
      "Step #2 - \"test_pipeline\": OK\n",
      "Finished Step #2 - \"test_pipeline\"\n",
      "Starting Step #3 - \"upload\"\n",
      "Step #3 - \"upload\": Already have image (with digest): gcr.io/cloud-builders/gsutil\n",
      "Step #3 - \"upload\": Copying file://pipeline.yaml [Content-Type=application/octet-stream]...\n",
      "/ [1 files][  3.8 KiB/  3.8 KiB]                                                                    \n",
      "Step #3 - \"upload\": Operation completed over 1 objects/3.8 KiB.\n",
      "Finished Step #3 - \"upload\"\n",
      "Starting Step #4 - \"test\"\n",
      "Step #4 - \"test\": Already have image: python\n",
      "Step #4 - \"test\": Submitting pipeline mlops-coaching-2-pipeline in experiment qa-mlops-coaching-2-experiment.\n",
      "Step #4 - \"test\": Creating PipelineJob\n",
      "Step #4 - \"test\": PipelineJob created. Resource name: projects/370018035372/locations/us-central1/pipelineJobs/mlops-coaching-pipeline-20230210104826\n",
      "Step #4 - \"test\": To use this PipelineJob in another session:\n",
      "Step #4 - \"test\": pipeline_job = aiplatform.PipelineJob.get('projects/370018035372/locations/us-central1/pipelineJobs/mlops-coaching-pipeline-20230210104826')\n",
      "Step #4 - \"test\": View Pipeline Job:\n",
      "Step #4 - \"test\": https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/mlops-coaching-pipeline-20230210104826?project=370018035372\n",
      "Step #4 - \"test\": Associating projects/370018035372/locations/us-central1/pipelineJobs/mlops-coaching-pipeline-20230210104826 to Experiment: qa-mlops-coaching-2-experiment\n",
      "Step #4 - \"test\": PipelineJob run completed. Resource name: projects/370018035372/locations/us-central1/pipelineJobs/mlops-coaching-pipeline-20230210104826\n",
      "Finished Step #4 - \"test\"\n",
      "Starting Step #5 - \"template\"\n",
      "Step #5 - \"template\": Already have image: python\n",
      "Finished Step #5 - \"template\"\n",
      "Starting Step #6 - \"prod\"\n",
      "Step #6 - \"prod\": Already have image: python\n",
      "Step #6 - \"prod\": Submitting pipeline mlops-coaching-2-pipeline in experiment prod-mlops-coaching-2-experiment.\n",
      "Step #6 - \"prod\": Creating PipelineJob\n",
      "Step #6 - \"prod\": PipelineJob created. Resource name: projects/370018035372/locations/us-central1/pipelineJobs/mlops-coaching-pipeline-20230210104843\n",
      "Step #6 - \"prod\": To use this PipelineJob in another session:\n",
      "Step #6 - \"prod\": pipeline_job = aiplatform.PipelineJob.get('projects/370018035372/locations/us-central1/pipelineJobs/mlops-coaching-pipeline-20230210104843')\n",
      "Step #6 - \"prod\": View Pipeline Job:\n",
      "Step #6 - \"prod\": https://console.cloud.google.com/vertex-ai/locations/us-central1/pipelines/runs/mlops-coaching-pipeline-20230210104843?project=370018035372\n",
      "Step #6 - \"prod\": Associating projects/370018035372/locations/us-central1/pipelineJobs/mlops-coaching-pipeline-20230210104843 to Experiment: prod-mlops-coaching-2-experiment\n",
      "Finished Step #6 - \"prod\"\n",
      "PUSH\n",
      "DONE\n",
      "--------------------------------------------------------------------------------\n",
      "ID                                    CREATE_TIME                DURATION  SOURCE                                                                                               IMAGES  STATUS\n",
      "599caf7a-a504-481d-bfb9-050191d41ee2  2023-02-10T10:47:17+00:00  1M30S     gs://vertex-ai-test-365213_cloudbuild/source/1676026036.796529-80b198a42908461cac0e8895c5563b37.tgz  -       SUCCESS\n"
     ]
    }
   ],
   "source": [
    "!gcloud builds submit ./src --config=src/cloudbuild.yaml --substitutions=_BUCKET_NAME=$BUCKET_NAME,_EXPERIMENT_NAME=$EXPERIMENT_NAME,_PIPELINE_NAME=$PIPELINE_NAME,_REGION=$REGION,_ENDPOINT_NAME=$ENDPOINT_NAME"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4c68abf9",
   "metadata": {},
   "source": [
    "### Create a git repository and trigger Cloud Build execution"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38f3a221",
   "metadata": {},
   "source": [
    "Before you can create the repository here, please **enable Cloud Source Repositories API** in the Google Cloud console.\n",
    "\n",
    "> <font color='green'>**Task 4**</font>\n",
    ">\n",
    "> Create a build trigger on the source repository that executes `cloudbuild.yaml`. Make sure to pass all **--substitutions**.\n",
    ">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "4953b256",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created [mlops-coaching-2-vertex-ai-test-365213].\n",
      "\u001b[1;33mWARNING:\u001b[0m You may be billed for this repository. See https://cloud.google.com/source-repositories/docs/pricing for details.\n"
     ]
    }
   ],
   "source": [
    "!gcloud source repos create $REPOSITORY_NAME"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "916f9b14",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created [https://cloudbuild.googleapis.com/v1/projects/vertex-ai-test-365213/locations/global/triggers/96b91f37-6821-4238-b0cb-d29db32dc931].\n",
      "NAME                   CREATE_TIME                STATUS\n",
      "mlops2-source-trigger  2023-02-10T10:49:24+00:00\n"
     ]
    }
   ],
   "source": [
    "!gcloud beta builds triggers create cloud-source-repositories \\\n",
    "    --name=mlops2-source-trigger \\\n",
    "    --repo=$REPOSITORY_NAME \\\n",
    "    --branch-pattern=master \\\n",
    "    --build-config=cloudbuild.yaml \\\n",
    "    --substitutions=_BUCKET_NAME=$BUCKET_NAME,_EXPERIMENT_NAME=$EXPERIMENT_NAME,_PIPELINE_NAME=$PIPELINE_NAME,_REGION=$REGION,_ENDPOINT_NAME=$ENDPOINT_NAME"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "6214fad2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Cloning into '/home/jupyter/dev-ml-coaching-2023-mlops/ml_coaching_part2/mlops-coaching-2-vertex-ai-test-365213'...\n",
      "warning: You appear to have cloned an empty repository.\n",
      "Project [vertex-ai-test-365213] repository [mlops-coaching-2-vertex-ai-test-365213] was cloned to [/home/jupyter/dev-ml-coaching-2023-mlops/ml_coaching_part2/mlops-coaching-2-vertex-ai-test-365213].\n"
     ]
    }
   ],
   "source": [
    "!gcloud source repos clone $REPOSITORY_NAME --project=$PROJECT_ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "a11fd771",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'src/__pycache__' -> 'mlops-coaching-2-vertex-ai-test-365213/__pycache__'\n",
      "'src/__pycache__/pipeline.cpython-37.pyc' -> 'mlops-coaching-2-vertex-ai-test-365213/__pycache__/pipeline.cpython-37.pyc'\n",
      "'src/cloudbuild.yaml' -> 'mlops-coaching-2-vertex-ai-test-365213/cloudbuild.yaml'\n",
      "'src/create-pipeline-template.py' -> 'mlops-coaching-2-vertex-ai-test-365213/create-pipeline-template.py'\n",
      "'src/pipeline.py' -> 'mlops-coaching-2-vertex-ai-test-365213/pipeline.py'\n",
      "'src/requirements.txt' -> 'mlops-coaching-2-vertex-ai-test-365213/requirements.txt'\n",
      "'src/submit-pipeline.py' -> 'mlops-coaching-2-vertex-ai-test-365213/submit-pipeline.py'\n",
      "'src/tests' -> 'mlops-coaching-2-vertex-ai-test-365213/tests'\n",
      "'src/tests/test_pipeline.py' -> 'mlops-coaching-2-vertex-ai-test-365213/tests/test_pipeline.py'\n",
      "'src/tests/__pycache__' -> 'mlops-coaching-2-vertex-ai-test-365213/tests/__pycache__'\n",
      "'src/tests/__pycache__/test_pipeline.cpython-37.pyc' -> 'mlops-coaching-2-vertex-ai-test-365213/tests/__pycache__/test_pipeline.cpython-37.pyc'\n"
     ]
    }
   ],
   "source": [
    "!cp -av src/* $REPOSITORY_NAME/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "dd844c57",
   "metadata": {},
   "outputs": [],
   "source": [
    "!cd $REPOSITORY_NAME && git add ."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "a09baca4",
   "metadata": {},
   "outputs": [],
   "source": [
    "!git config --global user.email \"{YOUR_EMAIl}\"\n",
    "!git config --global user.name \"{YOUR_NAME}\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "a067a2e7",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[master (root-commit) eab3406] initial commit\n",
      " 8 files changed, 153 insertions(+)\n",
      " create mode 100644 __pycache__/pipeline.cpython-37.pyc\n",
      " create mode 100644 cloudbuild.yaml\n",
      " create mode 100644 create-pipeline-template.py\n",
      " create mode 100644 pipeline.py\n",
      " create mode 100644 requirements.txt\n",
      " create mode 100644 submit-pipeline.py\n",
      " create mode 100644 tests/__pycache__/test_pipeline.cpython-37.pyc\n",
      " create mode 100644 tests/test_pipeline.py\n"
     ]
    }
   ],
   "source": [
    "!cd $REPOSITORY_NAME && git commit -a -m \"initial commit\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "6893f00e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Enumerating objects: 13, done.\n",
      "Counting objects: 100% (13/13), done.\n",
      "Delta compression using up to 4 threads\n",
      "Compressing objects: 100% (13/13), done.\n",
      "Writing objects: 100% (13/13), 3.89 KiB | 1.94 MiB/s, done.\n",
      "Total 13 (delta 0), reused 0 (delta 0), pack-reused 0\n",
      "To https://source.developers.google.com/p/vertex-ai-test-365213/r/mlops-coaching-2-vertex-ai-test-365213\n",
      " * [new branch]      master -> master\n",
      "branch 'master' set up to track 'origin/master'.\n"
     ]
    }
   ],
   "source": [
    "!cd $REPOSITORY_NAME && git push -u origin master"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1df96a6a-cdfd-495a-b95a-43ca2e2e142d",
   "metadata": {},
   "source": [
    "### Git - Cloud Build integration\n",
    "You will notice that a new Cloud Build pipeline was triggered from the commit you pushed. Any new commit to the repo will trigger a new pipeline being tested and deployed!\n",
    "Visit this link to see the [Cloud Build](https://console.cloud.google.com/cloud-build/builds) pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "999b8347",
   "metadata": {},
   "source": [
    "## Schedule pipeline execution\n",
    "\n",
    "> <font color='green'>**Task 5**</font>\n",
    ">\n",
    "> Schedule the execution of the pipeline such that it runs every day.\n",
    ">"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8fe7a9b",
   "metadata": {},
   "source": [
    "### Schedule Method 1: Cloud Scheduler → Pub/Sub → Cloud Functions\n",
    "\n",
    "Using this method, you create a Pub/Sub topic that triggers a Cloud Function that triggers a Vertex AI Pipeline. Note that the service account the Cloud Function runs as needs access to Cloud Storage and Vertex AI."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "af9da449-cfd2-4bee-a0bb-1656fddfe671",
   "metadata": {},
   "source": [
    "### 1. Create a Pub/Sub topic\n",
    "The `trigger-mlops-2-pipeline` is the name of the new topic you are creating:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "8a16acf0",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Created topic [projects/vertex-ai-test-365213/topics/trigger-mlops-coaching-2-pipeline].\n"
     ]
    }
   ],
   "source": [
    "!gcloud pubsub topics create trigger-mlops-2-pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4a0a3f7d-9c52-4cb6-b0a8-38c262ea0658",
   "metadata": {},
   "source": [
    "### 2. Deploy the cloud function\n",
    "This Cloud Funciton will invoked by Pub/Sub, and will trigger the Vertex AI Piepline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "c66e5b97",
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir -p cf-trigger"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "74ed20d3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing cf-trigger/requirements.txt\n"
     ]
    }
   ],
   "source": [
    "%%writefile cf-trigger/requirements.txt\n",
    "kfp==2.0.0b9\n",
    "pytest==7.2.0\n",
    "google-cloud-aiplatform==1.20.0\n",
    "pytz==2022.7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "3c8b1c16",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overwriting cf-trigger/main.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile cf-trigger/main.py\n",
    "\n",
    "import os\n",
    "import base64\n",
    "import json\n",
    "from google.cloud import aiplatform\n",
    "\n",
    "REGION = os.environ[\"REGION\"]\n",
    "PIPELINE_NAME = os.environ[\"PIPELINE_NAME\"]\n",
    "PROJECT_ID = os.environ[\"PROJECT_ID\"]\n",
    "BUCKET_NAME = os.environ[\"BUCKET_NAME\"]\n",
    "\n",
    "\n",
    "def subscribe(event, context):\n",
    "    payload_message = base64.b64decode(event['data']).decode('utf-8')\n",
    "    print(payload_message)\n",
    "    payload_json = json.loads(payload_message)\n",
    "    pipeline_name = payload_json['pipeline_name']\n",
    "\n",
    "    aiplatform.init(project=PROJECT_ID, location=REGION)\n",
    "    \n",
    "    job = aiplatform.PipelineJob(\n",
    "        display_name=PIPELINE_NAME,\n",
    "        template_path=f'https://{REGION}-kfp.pkg.dev/{PROJECT_ID}/mlops2-repo/{pipeline_name}',\n",
    "        location=REGION,\n",
    "        project=PROJECT_ID,\n",
    "        enable_caching=False,\n",
    "        pipeline_root=f'gs://{BUCKET_NAME}'\n",
    "    )\n",
    "\n",
    "    job.submit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "id": "1b053fa0",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Deploying function (may take a while - up to 2 minutes)...⠹                    \n",
      "For Cloud Build Logs, visit: https://console.cloud.google.com/cloud-build/builds;region=us-central1/794c626f-5279-4266-8ec0-58d81dee7628?project=370018035372\n",
      "Deploying function (may take a while - up to 2 minutes)...done.                \n",
      "availableMemoryMb: 256\n",
      "buildId: 794c626f-5279-4266-8ec0-58d81dee7628\n",
      "buildName: projects/370018035372/locations/us-central1/builds/794c626f-5279-4266-8ec0-58d81dee7628\n",
      "dockerRegistry: CONTAINER_REGISTRY\n",
      "entryPoint: subscribe\n",
      "environmentVariables:\n",
      "  BUCKET_NAME: mlops-coaching-2-vertex-ai-test-365213\n",
      "  PIPELINE_NAME: mlops-coaching-2-pipeline\n",
      "  PROJECT_ID: vertex-ai-test-365213\n",
      "  REGION: us-central1\n",
      "eventTrigger:\n",
      "  eventType: google.pubsub.topic.publish\n",
      "  failurePolicy: {}\n",
      "  resource: projects/vertex-ai-test-365213/topics/trigger-mlops-coaching-2-pipeline\n",
      "  service: pubsub.googleapis.com\n",
      "ingressSettings: ALLOW_INTERNAL_AND_GCLB\n",
      "labels:\n",
      "  deployment-tool: cli-gcloud\n",
      "maxInstances: 3000\n",
      "name: projects/vertex-ai-test-365213/locations/us-central1/functions/mlops-coaching-2-function\n",
      "runtime: python37\n",
      "serviceAccountEmail: 370018035372-compute@developer.gserviceaccount.com\n",
      "sourceUploadUrl: https://storage.googleapis.com/uploads-65393602418.us-central1.cloudfunctions.appspot.com/2a617aba-3eae-4e91-a4ac-f8ce5d15a3fb.zip\n",
      "status: ACTIVE\n",
      "timeout: 60s\n",
      "updateTime: '2023-02-10T12:01:17.193Z'\n",
      "versionId: '4'\n"
     ]
    }
   ],
   "source": [
    "!gcloud functions deploy mlops-2-function \\\n",
    "--source=./cf-trigger \\\n",
    "--entry-point=subscribe \\\n",
    "--trigger-topic trigger-mlops-2-pipeline \\\n",
    "--runtime python37 \\\n",
    "--ingress-settings internal-and-gclb \\\n",
    "--set-env-vars REGION=$REGION,PIPELINE_NAME=$PIPELINE_NAME,PROJECT_ID=$PROJECT_ID,BUCKET_NAME=$BUCKET_NAME \\\n",
    "--service-account $PROJECT_NUM-compute@developer.gserviceaccount.com"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78819294-96fa-4639-80b6-ecbf47af066a",
   "metadata": {},
   "source": [
    "### 3. Create Cloud Scheduler\n",
    "A cloud scheduler to publish a new message to Pub/Sub:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "id": "c855c7dc-bced-4cdb-aa7f-5a44bbc993eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "parameters = {\"pipeline_name\": \"mlops-workshop-pipeline/latest\"}\n",
    "message_body = json.dumps(parameters)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "id": "1a0d60e6-0afe-43ce-98e7-f868d96a7718",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'{\"pipeline_name\": \"mlops-coaching-pipeline/latest\"}'"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "message_body"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "id": "f3032bb6-1d87-499b-9cc3-6e44ce26862f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "messageIds:\n",
      "- '6851316306787686'\n"
     ]
    }
   ],
   "source": [
    "!gcloud pubsub topics publish trigger-mlops-2-pipeline \\\n",
    "  --message='{message_body}'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "d16d4775",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "name: projects/vertex-ai-test-365213/locations/us-central1/jobs/mlops-coaching-training-pipleline\n",
      "pubsubTarget:\n",
      "  data: eyJwaXBlbGluZV9uYW1lIjogIm1sb3BzLWNvYWNoaW5nLXBpcGVsaW5lL2xhdGVzdCJ9\n",
      "  topicName: projects/vertex-ai-test-365213/topics/trigger-mlops-coaching-2-pipeline\n",
      "retryConfig:\n",
      "  maxBackoffDuration: 3600s\n",
      "  maxDoublings: 16\n",
      "  maxRetryDuration: 0s\n",
      "  minBackoffDuration: 5s\n",
      "schedule: 35 11 * * *\n",
      "state: ENABLED\n",
      "timeZone: Etc/UTC\n",
      "userUpdateTime: '2023-02-10T12:12:34Z'\n"
     ]
    }
   ],
   "source": [
    "!gcloud scheduler jobs create pubsub mlops-workshop-training-pipleline \\\n",
    "--schedule \"35 11 * * *\" \\\n",
    "--topic=trigger-mlops-2-pipeline \\\n",
    "--location=us-central1 \\\n",
    "--message-body='{message_body}'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "80aedfb6-e56c-44e2-acc2-3d939a1452f7",
   "metadata": {},
   "source": [
    "### Trigger pipeline from scheduler\n",
    "Now that you deployed your Cloud Scheduler, Pubsub Topic and Cloud Function you can visit the [Cloud Scheduler page](https://console.cloud.google.com/cloudscheduler).to see the scheduled pipeline trigger. You can also trigger this from the console by clicking on the 3 dots on the right under `Actions` and then click on `Force job run`.\n",
    "\n",
    "You can see the triggered pipeline directly from the [Vertex Pipelines page](https://console.cloud.google.com/vertex-ai/pipelines/runs).\n",
    "\n",
    "Additionally, the logs produced when triggering the pipeline can be inspected in the [Cloud Function page](https://console.cloud.google.com/functions/details/us-central1/mlops-2-function?env=gen1&tab=logs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b2113e3",
   "metadata": {},
   "source": [
    "### Coming soon: Vertex AI Scheduler (in private preview)\n",
    "\n",
    "\n",
    "The Vertex AI Schedule Service API is a new resource that lets you schedule ad hoc or recurring Vertex AI Pipeline runs. The goal of this service is to make scheduling a one off or recurring pipeline run simple, such that any Vertex AI user can quickly understand and leverage schedules to implement naive continuous training in their business.\n",
    "\n",
    "This new API will replace our previous guidance of using Cloud Scheduler with Cloud Functions to schedule a pipeline run. It may support additional Vertex AI resources in the future.\n",
    "\n",
    "To use the Schedules API during the Private Preview release, your project ID must be allowlisted in the `SCHEDULED_RUNS_TRUSTED_TESTER` group, you can use projects that you previously signed up with—for example. If you’re not in the trusted testers group, sign up by using this [sign-up form](https://docs.google.com/forms/d/e/1FAIpQLScDxABxIvqjeM_279dwTMmVfFBJD7qmW2leyU_ZBTYutJ62uA/viewform?usp=sf_link). After signing up, wait for a confirmation from the Vertex AI Pipelines team before continuing.\n",
    "\n",
    "\n",
    "Note that **Scheduler is currently only available via its REST interface and not the Vertex AI SDK**. We don't require the usage of the scheduler for this workshop session."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae81b04b-bf8f-4b2b-a682-82d9f573c210",
   "metadata": {},
   "source": [
    "## Congratulations, you completed the last notebook!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9f35dd7d-ba73-41c0-8168-88cd8c79bafd",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-8.m102",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-8:m102"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}