Experiments - Autolog 5 models' parameters and metrics.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "id": "ur8xi4C7S06n"
   },
   "outputs": [],
   "source": [
    "# Copyright 2023 Google LLC\n",
    "#\n",
    "# Licensed under the Apache License, Version 2.0 (the \"License\");\n",
    "# you may not use this file except in compliance with the License.\n",
    "# You may obtain a copy of the License at\n",
    "#\n",
    "#     https://www.apache.org/licenses/LICENSE-2.0\n",
    "#\n",
    "# Unless required by applicable law or agreed to in writing, software\n",
    "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
    "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
    "# See the License for the specific language governing permissions and\n",
    "# limitations under the License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JAPoU8Sm5E6e"
   },
   "source": [
    "# Vertex AI Experiments: Autologging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "24743cf4a1e1"
   },
   "source": [
    "**_NOTE_**: This notebook has been tested in the following environment:\n",
    "\n",
    "* Python version = 3.10.10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tvgnzT1CKxrO",
    "tags": []
   },
   "source": [
    "## Overview\n",
    "\n",
    "As part of the data science team, you want to try different modeling approaches during experimentation phase.To guarantee reproducibility, each approach has different parameters that you need to manually track This is a time consuming task. To address this challenge, Vertex AI SDK introduces autologging, a one-line code SDK capability which leverages MLflow to provide automatic metrics and parameters tracking associated with your  Vertex AI Experiments and experiment runs. Learn more about [Autologging data to an experiment run](https://cloud.google.com/vertex-ai/docs/experiments/autolog-data)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "d975e698c9a4"
   },
   "source": [
    "### Objective\n",
    "\n",
    "In this tutorial, you learn how to use `Vertex AI Autologging`.\n",
    "\n",
    "This tutorial uses the following Google Cloud ML services and resources:\n",
    "\n",
    "- Vertex AI Experiments\n",
    "\n",
    "The steps performed include:\n",
    "\n",
    "- Enable autologging in the Vertex AI SDK.\n",
    "- Train one scikit-learn model\n",
    "- Train one LightGBM model\n",
    "- Train three XGBoost models (setting different hyperparameters)\n",
    "- See all the resulting experiment runs with the evaluation metrics (accuracy, precision, recall, and F1-score) and parameters autologged to Vertex AI Experiments without setting any experiment run.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "08d289fa873f"
   },
   "source": [
    "### Dataset\n",
    "\n",
    "The dataset is the [UCI Car Evaluation dataset](https://archive-beta.ics.uci.edu/dataset/19/car+evaluation), which is derived from simple hierarchical decision model and it contains attributions to predict car evaluation class."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "aed92deeb4a0"
   },
   "source": [
    "### Costs\n",
    "\n",
    "This tutorial uses billable components of Google Cloud:\n",
    "\n",
    "* Vertex AI Experiments\n",
    "* Vertex AI Tensorboard\n",
    "* Cloud Storage\n",
    "\n",
    "Learn about [Vertex AI pricing](https://cloud.google.com/vertex-ai/pricing),\n",
    "and [Cloud Storage pricing](https://cloud.google.com/storage/pricing),\n",
    "and use the [Pricing Calculator](https://cloud.google.com/products/calculator/)\n",
    "to generate a cost estimate based on your projected usage."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "i7EUnXsZhAGF"
   },
   "source": [
    "## Installation\n",
    "\n",
    "Install the following packages required to execute this notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "id": "2b4ef9b72d43"
   },
   "outputs": [],
   "source": [
    "# Install the packages\n",
    "USER = \"\"\n",
    "! pip3 install {USER} --upgrade --q google-cloud-aiplatform tensorflow\n",
    "! pip3 install {USER} --upgrade --q pandas scikit-learn matplotlib category_encoders xgboost==1.7.5 lightgbm==3.3.5 mlflow\n",
    "! pip3 install {USER} --upgrade --q protobuf==3.20.3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "58707a750154"
   },
   "source": [
    "### Colab only: Uncomment the following cell to restart the kernel.\n",
    "Automatically restart kernel after installs so that your environment can access the new packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "O6T0JnTqnpEA"
   },
   "outputs": [],
   "source": [
    "# import IPython\n",
    "\n",
    "# app = IPython.Application.instance()\n",
    "# app.kernel.do_shutdown(True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BF1j6f9HApxa"
   },
   "source": [
    "## Setup\n",
    "\n",
    "### Define env variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "id": "oM1iC_MfAts1",
    "outputId": "372e78ec-fa0f-4e82-f199-b8bb0ce30a74"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Updated property [core/project].\n"
     ]
    }
   ],
   "source": [
    "# Set the project id\n",
    "PROJECT_ID = \"my-project-id\"  # @param {type:\"string\"}\n",
    "! gcloud config set project {PROJECT_ID}\n",
    "\n",
    "# Set your region\n",
    "REGION = \"europe-west4\"  # @param {type: \"string\"}\n",
    "\n",
    "# Set a bucket to store intermediate artifacts\n",
    "BUCKET_URI =f\"gs://experiments-demo-{PROJECT_ID}\"  # @param {type:\"string\"}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-EcIXiGsCePi"
   },
   "source": [
    "**Only if your bucket doesn't already exist**: Run the following cell to create your Cloud Storage bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "id": "NIq7R4HZCfIc"
   },
   "outputs": [],
   "source": [
    "# ! gsutil mb -l $REGION -p $PROJECT_ID $BUCKET_URI"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Suk_ZwEyPR1X"
   },
   "source": [
    "### Upload dataset to BigQuery, load it to a pandas dataframe\n",
    "\n",
    "1. Upload the [ UCI Car Evaluation dataset](https://archive-beta.ics.uci.edu/dataset/19/car+evaluation) (mentioned above) to a BigQuery table (**your turn**)\n",
    "2. Upload data from this table to a pandas dataframe.\n",
    "\n",
    "(you could also interact with the data in CSV format if you prefer, but then you'd need to do some minor changes to the code)\n",
    "\n",
    "#### 2. Upload BigQuery table to a pandas dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "id": "HNeNo37hQSIW"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The google.cloud.bigquery extension is already loaded. To reload it, use:\n",
      "  %reload_ext google.cloud.bigquery\n"
     ]
    }
   ],
   "source": [
    "%load_ext google.cloud.bigquery\n",
    "\n",
    "from google.cloud import bigquery\n",
    "client = bigquery.Client()\n",
    "\n",
    "## JUST FOR COLAB\n",
    "#from google.colab import data_table\n",
    "#data_table.enable_dataframe_formatter()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "id": "TbcgndESPQAf"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "3f3031edc92a4113b86f67725dc09fee",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Query is running:   0%|          |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6a86b5e7c2804d7bbb3013be228eaf90",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Downloading:   0%|          |"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%%bigquery cars_df --project $PROJECT_ID\n",
    "\n",
    "SELECT * FROM `experiments_workshop.car_eval_preprocessed` # @param {type: \"string\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "id": "Vs8hk1r4mZVq"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>buying</th>\n",
       "      <th>maint</th>\n",
       "      <th>doors</th>\n",
       "      <th>persons</th>\n",
       "      <th>lug_boot</th>\n",
       "      <th>safety</th>\n",
       "      <th>class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>vhigh</td>\n",
       "      <td>vhigh</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>small</td>\n",
       "      <td>low</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>vhigh</td>\n",
       "      <td>vhigh</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>med</td>\n",
       "      <td>low</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>vhigh</td>\n",
       "      <td>vhigh</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>big</td>\n",
       "      <td>low</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>vhigh</td>\n",
       "      <td>vhigh</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>small</td>\n",
       "      <td>low</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>vhigh</td>\n",
       "      <td>vhigh</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>med</td>\n",
       "      <td>low</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1723</th>\n",
       "      <td>low</td>\n",
       "      <td>low</td>\n",
       "      <td>5more</td>\n",
       "      <td>4</td>\n",
       "      <td>med</td>\n",
       "      <td>high</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1724</th>\n",
       "      <td>low</td>\n",
       "      <td>low</td>\n",
       "      <td>5more</td>\n",
       "      <td>4</td>\n",
       "      <td>big</td>\n",
       "      <td>high</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1725</th>\n",
       "      <td>low</td>\n",
       "      <td>low</td>\n",
       "      <td>5more</td>\n",
       "      <td>more</td>\n",
       "      <td>small</td>\n",
       "      <td>high</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1726</th>\n",
       "      <td>low</td>\n",
       "      <td>low</td>\n",
       "      <td>5more</td>\n",
       "      <td>more</td>\n",
       "      <td>med</td>\n",
       "      <td>high</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1727</th>\n",
       "      <td>low</td>\n",
       "      <td>low</td>\n",
       "      <td>5more</td>\n",
       "      <td>more</td>\n",
       "      <td>big</td>\n",
       "      <td>high</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1728 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     buying  maint  doors persons lug_boot safety  class\n",
       "0     vhigh  vhigh      2       2    small    low      0\n",
       "1     vhigh  vhigh      2       2      med    low      0\n",
       "2     vhigh  vhigh      2       2      big    low      0\n",
       "3     vhigh  vhigh      2       4    small    low      0\n",
       "4     vhigh  vhigh      2       4      med    low      0\n",
       "...     ...    ...    ...     ...      ...    ...    ...\n",
       "1723    low    low  5more       4      med   high      1\n",
       "1724    low    low  5more       4      big   high      1\n",
       "1725    low    low  5more    more    small   high      1\n",
       "1726    low    low  5more    more      med   high      1\n",
       "1727    low    low  5more    more      big   high      1\n",
       "\n",
       "[1728 rows x 7 columns]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cars_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "960505627ddf"
   },
   "source": [
    "### Import libraries\n",
    "\n",
    "Import the Vertex AI SDK to log experiments in Vertex AI Experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "id": "PyQmSRbKA8r-"
   },
   "outputs": [],
   "source": [
    "from google.cloud import aiplatform as vertex_ai"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2mwQMym-Rteu"
   },
   "source": [
    "### Helper functions\n",
    "\n",
    "To run experiments it is not uncommon to define experiment helpers, one per each modelling approach you plan to evaluate. Below you define the following experiment helpers:\n",
    "\n",
    "*   `train_sklearn_model`: A helper function to train a Decision Tree model using Sklearn.\n",
    "*   `train_xgboost_model`: A helper function to train a Decision Tree model using XGBoost.\n",
    "*   `train_LGBM_model`: A helper function to train a Decision Tree model using LightGBM.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "id": "T5tWyGgeRvXC"
   },
   "outputs": [],
   "source": [
    "def set_seed(seed: int):\n",
    "    \"\"\"\n",
    "    A function to set the seed for reproducibility.\n",
    "    Args:\n",
    "        seed: Seed to be set\n",
    "    Returns:\n",
    "        None\n",
    "    \"\"\"\n",
    "    import random\n",
    "\n",
    "    import numpy as np\n",
    "    import tensorflow as tf\n",
    "\n",
    "    random.seed(seed)\n",
    "    np.random.seed(seed)\n",
    "    tf.random.set_seed(seed)\n",
    "\n",
    "    \n",
    "import pandas as pd\n",
    "\n",
    "\n",
    "def train_xgboost_model(df: pd.DataFrame, test_size: int, max_depth: int, n_estimators: int,  enable_categorical: bool=True):\n",
    "    \"\"\"\n",
    "    A function to train a Decision Tree model using XGboost.\n",
    "    Args:\n",
    "        df: Input data, which needs to be split into train and test\n",
    "        test_size: Size of the test set\n",
    "        max_depth: Maximum depth of the Decision Tree\n",
    "        n_estimators: Number of trees\n",
    "        enable_categorical: Whether to enable categorical features\n",
    "    Returns:\n",
    "        None\n",
    "    \"\"\"\n",
    "\n",
    "    # Libraries\n",
    "    \n",
    "    from xgboost import XGBClassifier\n",
    "    from sklearn import metrics\n",
    "    from sklearn.model_selection import train_test_split\n",
    "    from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "\n",
    "    # Convert categorical columns to numerical values\n",
    "    for column in cars_df.select_dtypes(include='object').columns:\n",
    "        label_encoder = LabelEncoder()\n",
    "        cars_df[column] = label_encoder.fit_transform(cars_df[column])\n",
    "\n",
    "    # Train, test split\n",
    "    print(\"Generating train and test data...\")\n",
    "    x = cars_df[[\"buying\", \"maint\", \"doors\", \"persons\", \"lug_boot\", \"safety\"]]\n",
    "    y = cars_df[[\"class\"]]\n",
    "    X_train, X_test, y_train, y_test = train_test_split(\n",
    "        x, y, test_size=test_size, shuffle=True\n",
    "    )\n",
    "\n",
    "    # Train model\n",
    "    print(\"Training model...\")\n",
    "    xgb_clf = XGBClassifier(\n",
    "        max_depth=max_depth, n_estimators=n_estimators, objective=\"binary:logistic\"\n",
    "    )\n",
    "    xgb_clf.fit(X_train, y_train)\n",
    "\n",
    "    # Evaluate model\n",
    "    print(\"Evaluating model...\")\n",
    "    y_pred = xgb_clf.predict(X_test)\n",
    "    accuracy = metrics.accuracy_score(y_test, y_pred)\n",
    "    print(\"Accuracy:\", round(accuracy, 3))\n",
    "    precision = metrics.precision_score(y_test, y_pred)\n",
    "    print(\"Precision:\", round(precision, 3))\n",
    "    recall = metrics.recall_score(y_test, y_pred)\n",
    "    print(\"Recall:\", round(recall, 3))\n",
    "    f1 = metrics.f1_score(y_test, y_pred)\n",
    "    print(\"F1 score:\", round(f1, 3))\n",
    "\n",
    "\n",
    "def train_sklearn_model(df: pd.DataFrame, test_size: int, max_depth: int, enable_categorical: bool=True):\n",
    "    \"\"\"\n",
    "    A function to train a Decision Tree model using scikit-learn.\n",
    "    Args:\n",
    "        df: Input data, which needs to be split into train and test\n",
    "        test_size: Size of the test set\n",
    "        max_depth: Maximum depth of the Decision Tree\n",
    "        n_estimators: Number of trees\n",
    "        enable_categorical: Whether to enable categorical features\n",
    "    Returns:\n",
    "        None\n",
    "    \"\"\"\n",
    "\n",
    "    # Libraries\n",
    "    from sklearn.tree import DecisionTreeClassifier\n",
    "    from sklearn import metrics\n",
    "    from sklearn.model_selection import train_test_split\n",
    "    from sklearn.preprocessing import LabelEncoder\n",
    "    import matplotlib\n",
    "\n",
    "    # Convert categorical columns to numerical values\n",
    "    for column in cars_df.select_dtypes(include='object').columns:\n",
    "        label_encoder = LabelEncoder()\n",
    "        cars_df[column] = label_encoder.fit_transform(cars_df[column])\n",
    "\n",
    "    # Train, test split\n",
    "    print(\"Generating train and test data...\")\n",
    "    x = cars_df[[\"buying\", \"maint\", \"doors\", \"persons\", \"lug_boot\", \"safety\"]]\n",
    "    y = cars_df[[\"class\"]]\n",
    "    X_train, X_test, y_train, y_test = train_test_split(\n",
    "        x, y, test_size=test_size, shuffle=True\n",
    "    )\n",
    "\n",
    "    # Train model\n",
    "    print(\"Training model...\")\n",
    "    dt_clf = DecisionTreeClassifier(max_depth=max_depth)\n",
    "    dt_clf.fit(X_train, y_train)\n",
    "\n",
    "    # Evaluate model\n",
    "    print(\"Evaluating model...\")\n",
    "    y_pred = dt_clf.predict(X_test)\n",
    "    accuracy = metrics.accuracy_score(y_test, y_pred)\n",
    "    print(\"Accuracy:\", round(accuracy, 3))\n",
    "    precision = metrics.precision_score(y_test, y_pred)\n",
    "    print(\"Precision:\", round(precision, 3))\n",
    "    recall = metrics.recall_score(y_test, y_pred)\n",
    "    print(\"Recall:\", round(recall, 3))\n",
    "    f1 = metrics.f1_score(y_test, y_pred)\n",
    "    print(\"F1 score:\", round(f1, 3))\n",
    "    \n",
    "\n",
    "def train_lgbm_model(df, test_size=0.25, max_depth=3, n_estimators=100):\n",
    "    \"\"\"\n",
    "    A function to train a Light GBM model.\n",
    "    Args:\n",
    "        df: Input data, which needs to be split into train and test\n",
    "        test_size: Size of the test set\n",
    "        max_depth: Maximum depth of the Decision Tree\n",
    "        n_estimators: Number of trees\n",
    "        enable_categorical: Whether to enable categorical features\n",
    "    Returns:\n",
    "        None\n",
    "    \"\"\"\n",
    "\n",
    "    # Libraries\n",
    "    import lightgbm as lgb\n",
    "    from sklearn import metrics\n",
    "    from sklearn.metrics import accuracy_score\n",
    "    from sklearn.model_selection import train_test_split\n",
    "    from sklearn.preprocessing import LabelEncoder\n",
    "\n",
    "    # Convert categorical columns to numerical values\n",
    "    for column in cars_df.select_dtypes(include='object').columns:\n",
    "        label_encoder = LabelEncoder()\n",
    "        cars_df[column] = label_encoder.fit_transform(cars_df[column])\n",
    "\n",
    "    # Train, test split\n",
    "    print(\"Generating train and test data...\")\n",
    "    x = cars_df[[\"buying\", \"maint\", \"doors\", \"persons\", \"lug_boot\", \"safety\"]]\n",
    "    y = cars_df[[\"class\"]]\n",
    "    X_train, X_test, y_train, y_test = train_test_split(\n",
    "        x, y, test_size=test_size, shuffle=True\n",
    "    )\n",
    "\n",
    "    # Create a label encoder for the target variable\n",
    "    label_encoder = LabelEncoder()\n",
    "    y_train = label_encoder.fit_transform(y_train)\n",
    "    y_test = label_encoder.transform(y_test)\n",
    "\n",
    "    # Create a LightGBM model\n",
    "    model = lgb.LGBMClassifier(objective='binary', max_depth=max_depth, n_estimators=n_estimators)\n",
    "\n",
    "    # Train the model on the train set\n",
    "    print(\"Training model...\")\n",
    "    model.fit(X_train, y_train)\n",
    "\n",
    "    # Evaluate the model on the test set\n",
    "    y_pred = model.predict(X_test)\n",
    "    accuracy = metrics.accuracy_score(y_test, y_pred)\n",
    "    print(\"Accuracy:\", round(accuracy, 3))\n",
    "    precision = metrics.precision_score(y_test, y_pred)\n",
    "    print(\"Precision:\", round(precision, 3))\n",
    "    recall = metrics.recall_score(y_test, y_pred)\n",
    "    print(\"Recall:\", round(recall, 3))\n",
    "    f1 = metrics.f1_score(y_test, y_pred)\n",
    "    print(\"F1 score:\", round(f1, 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "004aGQ1BSLFm"
   },
   "source": [
    "## Model experimentation using autologging with Vertex AI Experiments\n",
    "\n",
    "Vertex AI Experiments Autologging allows you to run experiments and autologging parameters and metrics of different ML frameworks.\n",
    "\n",
    "After initiating an Vertex AI Experiment, enable autologging using `vertex_ai.autolog()`.\n",
    "\n",
    "There are two ways to use Autologging:\n",
    "\n",
    "1.   *With automatic experiment run creation*\n",
    "2.   *With user experiment run creation*\n",
    "\n",
    "With *automatic experiment run creation*, you run an experiment. Vertex AI SDK automatically creates an experiment run by logging all paramenters and metrics in Vertex AI Experiments.\n",
    "\n",
    "With *user experiment run creation*, you create an experiment using `vertex_ai.start_run(your-experiment-run-name)` and run the experiment. Then you get access to resulting paramentes and metrics after you end the experiment run with `vertex_ai.end_run()`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JjJVARe5V-MG"
   },
   "source": [
    "#### Create an experiment for tracking training parameters and metrics\n",
    "\n",
    "To start, initiate an experiment using the `init()` method.\n",
    "\n",
    "Because some model types like TensorFlow result in autologging time series metrics, you need to create a TensorBoard instance.\n",
    "\n",
    "To create a TensorBoard instance, you can use `vertex_ai.Tensorboard.create()`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mRoiPPk1XCWi"
   },
   "source": [
    " ⚠⚠⚠ **Warning** Currently, Vertex AI TensorBoard charges a monthly fee of 300 USD per unique active user. Beginning in August 2023, Vertex AI TensorBoard will change from a per-user monthly license of 300 USD per month to 10 USD per GiB per month for storage of logs and metrics. This means there are no more subscription fees, and you will only pay for the storage you've used. See the [Vertex AI TensorBoard: Delete Outdated TensorBoard Experiments tutorial](https://github.com/GoogleCloudPlatform/vertex-ai-samples/blob/main/notebooks/official/experiments/delete_outdated_tensorboard_experiments.ipynb) for how to manage storage. More info on Tensorboard [here](https://cloud.google.com/vertex-ai/docs/experiments/tensorboard-overview).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "r2z-VaIVTtI5"
   },
   "outputs": [],
   "source": [
    "AUTOLOGGED_EXPERIMENT_NAME = \"my-experiment\"  # @param {type:\"string\"}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "Trjm4Dq3yH_o"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating Tensorboard\n",
      "Create Tensorboard backing LRO: projects/1080686785400/locations/us-central1/tensorboards/3902149174790979584/operations/28639185527111680\n",
      "Tensorboard created. Resource name: projects/1080686785400/locations/us-central1/tensorboards/3902149174790979584\n",
      "To use this Tensorboard in another session:\n",
      "tb = aiplatform.Tensorboard('projects/1080686785400/locations/us-central1/tensorboards/3902149174790979584')\n"
     ]
    }
   ],
   "source": [
    "EXPERIMENT_TENSORBOARD = vertex_ai.Tensorboard.create()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "id": "eqM2pIO8_4Y7"
   },
   "outputs": [],
   "source": [
    "vertex_ai.init(\n",
    "    project=PROJECT_ID,\n",
    "    location=REGION,\n",
    "    staging_bucket=BUCKET_URI,\n",
    "    experiment=AUTOLOGGED_EXPERIMENT_NAME,\n",
    "    experiment_tensorboard=EXPERIMENT_TENSORBOARD,\n",
    "    experiment_description=\"autolog-experiment-with-automatic-run\",\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "LCoYnaY594V7"
   },
   "source": [
    "#### Autologging an experiment with automatic experiment run creation\n",
    "\n",
    "In this section, Vertex AI SDK automatically creates an experiment run for you by logging all paramenters and training and post-training metrics in Vertex AI Experiments.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8VGUnqk6Zvb6"
   },
   "source": [
    "##### Enable autologging\n",
    "\n",
    "First, enable autologging using `vertex_ai.autolog()` method.\n",
    "\n",
    "After calling `vertex_ai.autolog()`, any metrics and parameters from\n",
    "model training calls with supported ML frameworks will be automatically\n",
    "logged to Vertex Experiments."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "k0H9kw74-5ob"
   },
   "outputs": [],
   "source": [
    "vertex_ai.autolog()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yWWr08gZZzMb"
   },
   "source": [
    "#### Experiment 1 - Sklearn model\n",
    "\n",
    "Next, define your baseline model by running a Sklearn model experiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "aLCVSnA_zbpN"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating train and test data...\n",
      "Training model...\n",
      "Associating projects/1080686785400/locations/europe-west4/metadataStores/default/contexts/my-experiment-sklearn-2023-07-27-18-57-13-0f510 to Experiment: my-experiment\n",
      "Evaluating model...\n",
      "Accuracy: 0.948\n",
      "Precision: 0.71\n",
      "Recall: 0.71\n",
      "F1 score: 0.71\n"
     ]
    }
   ],
   "source": [
    "train_sklearn_model(cars_df, test_size=0.2, max_depth=5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cQh3v_85aGWP"
   },
   "source": [
    "##### Get the experiment results\n",
    "\n",
    "Then, use the method `get_experiment_df()` to get the results of the experiment as a pandas dataframe.\n",
    "\n",
    "Notice how all paramenters and metrics are logged in Vertex AI Experiments.\n",
    "\n",
    "In particular, the `run_name` has been automatically assigned and the evaluation metrics you defined have been logged too."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "id": "KuAl1etkLwAe"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>...</th>\n",
       "      <th>14</th>\n",
       "      <th>15</th>\n",
       "      <th>16</th>\n",
       "      <th>17</th>\n",
       "      <th>18</th>\n",
       "      <th>19</th>\n",
       "      <th>20</th>\n",
       "      <th>21</th>\n",
       "      <th>22</th>\n",
       "      <th>23</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>experiment_name</th>\n",
       "      <td>my-experiment</td>\n",
       "      <td>my-experiment</td>\n",
       "      <td>my-experiment</td>\n",
       "      <td>my-experiment</td>\n",
       "      <td>my-experiment</td>\n",
       "      <td>my-experiment</td>\n",
       "      <td>my-experiment</td>\n",
       "      <td>my-experiment</td>\n",
       "      <td>my-experiment</td>\n",
       "      <td>my-experiment</td>\n",
       "      <td>...</td>\n",
       "      <td>my-experiment</td>\n",
       "      <td>my-experiment</td>\n",
       "      <td>my-experiment</td>\n",
       "      <td>my-experiment</td>\n",
       "      <td>my-experiment</td>\n",
       "      <td>my-experiment</td>\n",
       "      <td>my-experiment</td>\n",
       "      <td>my-experiment</td>\n",
       "      <td>my-experiment</td>\n",
       "      <td>my-experiment</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>run_name</th>\n",
       "      <td>sklearn-2023-07-27-18-57-13-0f510</td>\n",
       "      <td>lightgbm-2023-07-27-18-43-42-6a461</td>\n",
       "      <td>sklearn-2023-07-27-18-35-28-9337e</td>\n",
       "      <td>sklearn-2023-07-26-12-59-42-89c02</td>\n",
       "      <td>sklearn-2023-07-26-12-54-00-35fc5</td>\n",
       "      <td>sklearn-2023-07-26-12-34-34-f191b</td>\n",
       "      <td>sklearn-2023-07-26-12-34-06-f5757</td>\n",
       "      <td>sklearn-2023-07-26-12-33-03-d4572</td>\n",
       "      <td>sklearn-2023-07-26-12-31-13-0eb6a</td>\n",
       "      <td>sklearn-2023-07-26-12-28-01-05c79</td>\n",
       "      <td>...</td>\n",
       "      <td>xgboost-2023-07-26-12-10-41-e7f18</td>\n",
       "      <td>lightgbm-2023-07-26-12-10-31-1cbc7</td>\n",
       "      <td>sklearn-2023-07-26-12-10-02-e6da6</td>\n",
       "      <td>sklearn-2023-07-26-12-04-27-3195e</td>\n",
       "      <td>sklearn-2023-07-26-12-02-34-99200</td>\n",
       "      <td>xgboost-2023-07-26-11-40-22-71be7</td>\n",
       "      <td>xgboost-2023-07-26-11-40-13-7e36e</td>\n",
       "      <td>xgboost-2023-07-26-11-39-58-cd7a4</td>\n",
       "      <td>lightgbm-2023-07-26-11-35-23-83821</td>\n",
       "      <td>sklearn-2023-07-26-11-35-08-cbf96</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>run_type</th>\n",
       "      <td>system.ExperimentRun</td>\n",
       "      <td>system.ExperimentRun</td>\n",
       "      <td>system.ExperimentRun</td>\n",
       "      <td>system.ExperimentRun</td>\n",
       "      <td>system.ExperimentRun</td>\n",
       "      <td>system.ExperimentRun</td>\n",
       "      <td>system.ExperimentRun</td>\n",
       "      <td>system.ExperimentRun</td>\n",
       "      <td>system.ExperimentRun</td>\n",
       "      <td>system.ExperimentRun</td>\n",
       "      <td>...</td>\n",
       "      <td>system.ExperimentRun</td>\n",
       "      <td>system.ExperimentRun</td>\n",
       "      <td>system.ExperimentRun</td>\n",
       "      <td>system.ExperimentRun</td>\n",
       "      <td>system.ExperimentRun</td>\n",
       "      <td>system.ExperimentRun</td>\n",
       "      <td>system.ExperimentRun</td>\n",
       "      <td>system.ExperimentRun</td>\n",
       "      <td>system.ExperimentRun</td>\n",
       "      <td>system.ExperimentRun</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>state</th>\n",
       "      <td>COMPLETE</td>\n",
       "      <td>COMPLETE</td>\n",
       "      <td>COMPLETE</td>\n",
       "      <td>COMPLETE</td>\n",
       "      <td>COMPLETE</td>\n",
       "      <td>COMPLETE</td>\n",
       "      <td>COMPLETE</td>\n",
       "      <td>COMPLETE</td>\n",
       "      <td>COMPLETE</td>\n",
       "      <td>COMPLETE</td>\n",
       "      <td>...</td>\n",
       "      <td>COMPLETE</td>\n",
       "      <td>COMPLETE</td>\n",
       "      <td>COMPLETE</td>\n",
       "      <td>COMPLETE</td>\n",
       "      <td>COMPLETE</td>\n",
       "      <td>COMPLETE</td>\n",
       "      <td>COMPLETE</td>\n",
       "      <td>COMPLETE</td>\n",
       "      <td>COMPLETE</td>\n",
       "      <td>COMPLETE</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>param.class_weight</th>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>None</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>metric.recall_score_X_test</th>\n",
       "      <td>NaN</td>\n",
       "      <td>0.72</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>0.928571</td>\n",
       "      <td>0.75</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.75</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>metric.precision_score_x_test</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.941176</td>\n",
       "      <td>0.925926</td>\n",
       "      <td>0.764706</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>metric.recall_score_x_test</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.961538</td>\n",
       "      <td>0.928571</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>metric.accuracy_score_x_test</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.99711</td>\n",
       "      <td>0.991329</td>\n",
       "      <td>0.971098</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>metric.f1_score_x_test</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0.969697</td>\n",
       "      <td>0.943396</td>\n",
       "      <td>0.83871</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>95 rows × 24 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                              0   \\\n",
       "experiment_name                                    my-experiment   \n",
       "run_name                       sklearn-2023-07-27-18-57-13-0f510   \n",
       "run_type                                    system.ExperimentRun   \n",
       "state                                                   COMPLETE   \n",
       "param.class_weight                                          None   \n",
       "...                                                          ...   \n",
       "metric.recall_score_X_test                                   NaN   \n",
       "metric.precision_score_x_test                                NaN   \n",
       "metric.recall_score_x_test                                   NaN   \n",
       "metric.accuracy_score_x_test                                 NaN   \n",
       "metric.f1_score_x_test                                       NaN   \n",
       "\n",
       "                                                               1   \\\n",
       "experiment_name                                     my-experiment   \n",
       "run_name                       lightgbm-2023-07-27-18-43-42-6a461   \n",
       "run_type                                     system.ExperimentRun   \n",
       "state                                                    COMPLETE   \n",
       "param.class_weight                                            NaN   \n",
       "...                                                           ...   \n",
       "metric.recall_score_X_test                                   0.72   \n",
       "metric.precision_score_x_test                                 NaN   \n",
       "metric.recall_score_x_test                                    NaN   \n",
       "metric.accuracy_score_x_test                                  NaN   \n",
       "metric.f1_score_x_test                                        NaN   \n",
       "\n",
       "                                                              2   \\\n",
       "experiment_name                                    my-experiment   \n",
       "run_name                       sklearn-2023-07-27-18-35-28-9337e   \n",
       "run_type                                    system.ExperimentRun   \n",
       "state                                                   COMPLETE   \n",
       "param.class_weight                                          None   \n",
       "...                                                          ...   \n",
       "metric.recall_score_X_test                                   NaN   \n",
       "metric.precision_score_x_test                                NaN   \n",
       "metric.recall_score_x_test                                   NaN   \n",
       "metric.accuracy_score_x_test                                 NaN   \n",
       "metric.f1_score_x_test                                       NaN   \n",
       "\n",
       "                                                              3   \\\n",
       "experiment_name                                    my-experiment   \n",
       "run_name                       sklearn-2023-07-26-12-59-42-89c02   \n",
       "run_type                                    system.ExperimentRun   \n",
       "state                                                   COMPLETE   \n",
       "param.class_weight                                           NaN   \n",
       "...                                                          ...   \n",
       "metric.recall_score_X_test                                   NaN   \n",
       "metric.precision_score_x_test                                NaN   \n",
       "metric.recall_score_x_test                                   NaN   \n",
       "metric.accuracy_score_x_test                                 NaN   \n",
       "metric.f1_score_x_test                                       NaN   \n",
       "\n",
       "                                                              4   \\\n",
       "experiment_name                                    my-experiment   \n",
       "run_name                       sklearn-2023-07-26-12-54-00-35fc5   \n",
       "run_type                                    system.ExperimentRun   \n",
       "state                                                   COMPLETE   \n",
       "param.class_weight                                           NaN   \n",
       "...                                                          ...   \n",
       "metric.recall_score_X_test                                   NaN   \n",
       "metric.precision_score_x_test                                NaN   \n",
       "metric.recall_score_x_test                                   NaN   \n",
       "metric.accuracy_score_x_test                                 NaN   \n",
       "metric.f1_score_x_test                                       NaN   \n",
       "\n",
       "                                                              5   \\\n",
       "experiment_name                                    my-experiment   \n",
       "run_name                       sklearn-2023-07-26-12-34-34-f191b   \n",
       "run_type                                    system.ExperimentRun   \n",
       "state                                                   COMPLETE   \n",
       "param.class_weight                                           NaN   \n",
       "...                                                          ...   \n",
       "metric.recall_score_X_test                                   NaN   \n",
       "metric.precision_score_x_test                                NaN   \n",
       "metric.recall_score_x_test                                   NaN   \n",
       "metric.accuracy_score_x_test                                 NaN   \n",
       "metric.f1_score_x_test                                       NaN   \n",
       "\n",
       "                                                              6   \\\n",
       "experiment_name                                    my-experiment   \n",
       "run_name                       sklearn-2023-07-26-12-34-06-f5757   \n",
       "run_type                                    system.ExperimentRun   \n",
       "state                                                   COMPLETE   \n",
       "param.class_weight                                           NaN   \n",
       "...                                                          ...   \n",
       "metric.recall_score_X_test                                   NaN   \n",
       "metric.precision_score_x_test                                NaN   \n",
       "metric.recall_score_x_test                                   NaN   \n",
       "metric.accuracy_score_x_test                                 NaN   \n",
       "metric.f1_score_x_test                                       NaN   \n",
       "\n",
       "                                                              7   \\\n",
       "experiment_name                                    my-experiment   \n",
       "run_name                       sklearn-2023-07-26-12-33-03-d4572   \n",
       "run_type                                    system.ExperimentRun   \n",
       "state                                                   COMPLETE   \n",
       "param.class_weight                                           NaN   \n",
       "...                                                          ...   \n",
       "metric.recall_score_X_test                                   NaN   \n",
       "metric.precision_score_x_test                                NaN   \n",
       "metric.recall_score_x_test                                   NaN   \n",
       "metric.accuracy_score_x_test                                 NaN   \n",
       "metric.f1_score_x_test                                       NaN   \n",
       "\n",
       "                                                              8   \\\n",
       "experiment_name                                    my-experiment   \n",
       "run_name                       sklearn-2023-07-26-12-31-13-0eb6a   \n",
       "run_type                                    system.ExperimentRun   \n",
       "state                                                   COMPLETE   \n",
       "param.class_weight                                           NaN   \n",
       "...                                                          ...   \n",
       "metric.recall_score_X_test                                   NaN   \n",
       "metric.precision_score_x_test                                NaN   \n",
       "metric.recall_score_x_test                                   NaN   \n",
       "metric.accuracy_score_x_test                                 NaN   \n",
       "metric.f1_score_x_test                                       NaN   \n",
       "\n",
       "                                                              9   ...  \\\n",
       "experiment_name                                    my-experiment  ...   \n",
       "run_name                       sklearn-2023-07-26-12-28-01-05c79  ...   \n",
       "run_type                                    system.ExperimentRun  ...   \n",
       "state                                                   COMPLETE  ...   \n",
       "param.class_weight                                           NaN  ...   \n",
       "...                                                          ...  ...   \n",
       "metric.recall_score_X_test                                   NaN  ...   \n",
       "metric.precision_score_x_test                                NaN  ...   \n",
       "metric.recall_score_x_test                                   NaN  ...   \n",
       "metric.accuracy_score_x_test                                 NaN  ...   \n",
       "metric.f1_score_x_test                                       NaN  ...   \n",
       "\n",
       "                                                              14  \\\n",
       "experiment_name                                    my-experiment   \n",
       "run_name                       xgboost-2023-07-26-12-10-41-e7f18   \n",
       "run_type                                    system.ExperimentRun   \n",
       "state                                                   COMPLETE   \n",
       "param.class_weight                                           NaN   \n",
       "...                                                          ...   \n",
       "metric.recall_score_X_test                              0.928571   \n",
       "metric.precision_score_x_test                                NaN   \n",
       "metric.recall_score_x_test                                   NaN   \n",
       "metric.accuracy_score_x_test                                 NaN   \n",
       "metric.f1_score_x_test                                       NaN   \n",
       "\n",
       "                                                               15  \\\n",
       "experiment_name                                     my-experiment   \n",
       "run_name                       lightgbm-2023-07-26-12-10-31-1cbc7   \n",
       "run_type                                     system.ExperimentRun   \n",
       "state                                                    COMPLETE   \n",
       "param.class_weight                                            NaN   \n",
       "...                                                           ...   \n",
       "metric.recall_score_X_test                                   0.75   \n",
       "metric.precision_score_x_test                                 NaN   \n",
       "metric.recall_score_x_test                                    NaN   \n",
       "metric.accuracy_score_x_test                                  NaN   \n",
       "metric.f1_score_x_test                                        NaN   \n",
       "\n",
       "                                                              16  \\\n",
       "experiment_name                                    my-experiment   \n",
       "run_name                       sklearn-2023-07-26-12-10-02-e6da6   \n",
       "run_type                                    system.ExperimentRun   \n",
       "state                                                   COMPLETE   \n",
       "param.class_weight                                           NaN   \n",
       "...                                                          ...   \n",
       "metric.recall_score_X_test                                   NaN   \n",
       "metric.precision_score_x_test                                NaN   \n",
       "metric.recall_score_x_test                                   NaN   \n",
       "metric.accuracy_score_x_test                                 NaN   \n",
       "metric.f1_score_x_test                                       NaN   \n",
       "\n",
       "                                                              17  \\\n",
       "experiment_name                                    my-experiment   \n",
       "run_name                       sklearn-2023-07-26-12-04-27-3195e   \n",
       "run_type                                    system.ExperimentRun   \n",
       "state                                                   COMPLETE   \n",
       "param.class_weight                                           NaN   \n",
       "...                                                          ...   \n",
       "metric.recall_score_X_test                                   NaN   \n",
       "metric.precision_score_x_test                                NaN   \n",
       "metric.recall_score_x_test                                   NaN   \n",
       "metric.accuracy_score_x_test                                 NaN   \n",
       "metric.f1_score_x_test                                       NaN   \n",
       "\n",
       "                                                              18  \\\n",
       "experiment_name                                    my-experiment   \n",
       "run_name                       sklearn-2023-07-26-12-02-34-99200   \n",
       "run_type                                    system.ExperimentRun   \n",
       "state                                                   COMPLETE   \n",
       "param.class_weight                                           NaN   \n",
       "...                                                          ...   \n",
       "metric.recall_score_X_test                                   NaN   \n",
       "metric.precision_score_x_test                                NaN   \n",
       "metric.recall_score_x_test                                   NaN   \n",
       "metric.accuracy_score_x_test                                 NaN   \n",
       "metric.f1_score_x_test                                       NaN   \n",
       "\n",
       "                                                              19  \\\n",
       "experiment_name                                    my-experiment   \n",
       "run_name                       xgboost-2023-07-26-11-40-22-71be7   \n",
       "run_type                                    system.ExperimentRun   \n",
       "state                                                   COMPLETE   \n",
       "param.class_weight                                           NaN   \n",
       "...                                                          ...   \n",
       "metric.recall_score_X_test                                   NaN   \n",
       "metric.precision_score_x_test                           0.941176   \n",
       "metric.recall_score_x_test                                   1.0   \n",
       "metric.accuracy_score_x_test                             0.99711   \n",
       "metric.f1_score_x_test                                  0.969697   \n",
       "\n",
       "                                                              20  \\\n",
       "experiment_name                                    my-experiment   \n",
       "run_name                       xgboost-2023-07-26-11-40-13-7e36e   \n",
       "run_type                                    system.ExperimentRun   \n",
       "state                                                   COMPLETE   \n",
       "param.class_weight                                           NaN   \n",
       "...                                                          ...   \n",
       "metric.recall_score_X_test                                   NaN   \n",
       "metric.precision_score_x_test                           0.925926   \n",
       "metric.recall_score_x_test                              0.961538   \n",
       "metric.accuracy_score_x_test                            0.991329   \n",
       "metric.f1_score_x_test                                  0.943396   \n",
       "\n",
       "                                                              21  \\\n",
       "experiment_name                                    my-experiment   \n",
       "run_name                       xgboost-2023-07-26-11-39-58-cd7a4   \n",
       "run_type                                    system.ExperimentRun   \n",
       "state                                                   COMPLETE   \n",
       "param.class_weight                                           NaN   \n",
       "...                                                          ...   \n",
       "metric.recall_score_X_test                                   NaN   \n",
       "metric.precision_score_x_test                           0.764706   \n",
       "metric.recall_score_x_test                              0.928571   \n",
       "metric.accuracy_score_x_test                            0.971098   \n",
       "metric.f1_score_x_test                                   0.83871   \n",
       "\n",
       "                                                               22  \\\n",
       "experiment_name                                     my-experiment   \n",
       "run_name                       lightgbm-2023-07-26-11-35-23-83821   \n",
       "run_type                                     system.ExperimentRun   \n",
       "state                                                    COMPLETE   \n",
       "param.class_weight                                            NaN   \n",
       "...                                                           ...   \n",
       "metric.recall_score_X_test                                   0.75   \n",
       "metric.precision_score_x_test                                 NaN   \n",
       "metric.recall_score_x_test                                    NaN   \n",
       "metric.accuracy_score_x_test                                  NaN   \n",
       "metric.f1_score_x_test                                        NaN   \n",
       "\n",
       "                                                              23  \n",
       "experiment_name                                    my-experiment  \n",
       "run_name                       sklearn-2023-07-26-11-35-08-cbf96  \n",
       "run_type                                    system.ExperimentRun  \n",
       "state                                                   COMPLETE  \n",
       "param.class_weight                                           NaN  \n",
       "...                                                          ...  \n",
       "metric.recall_score_X_test                                   NaN  \n",
       "metric.precision_score_x_test                                NaN  \n",
       "metric.recall_score_x_test                                   NaN  \n",
       "metric.accuracy_score_x_test                                 NaN  \n",
       "metric.f1_score_x_test                                       NaN  \n",
       "\n",
       "[95 rows x 24 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "experiment_df = vertex_ai.get_experiment_df()\n",
    "experiment_df = experiment_df.T\n",
    "experiment_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qQlwVV_VMjGZ"
   },
   "source": [
    "#### Experiment 2 - LGBM model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "id": "qOu7DxpVVMMY"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating train and test data...\n",
      "Training model...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/opt/conda/lib/python3.10/site-packages/sklearn/preprocessing/_label.py:114: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, warn=True)\n",
      "/opt/conda/lib/python3.10/site-packages/sklearn/preprocessing/_label.py:132: DataConversionWarning: A column-vector y was passed when a 1d array was expected. Please change the shape of y to (n_samples, ), for example using ravel().\n",
      "  y = column_or_1d(y, dtype=self.classes_.dtype, warn=True)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Associating projects/1080686785400/locations/europe-west4/metadataStores/default/contexts/my-experiment-lightgbm-2023-07-27-18-58-13-a66d9 to Experiment: my-experiment\n",
      "Accuracy: 0.957\n",
      "Precision: 0.842\n",
      "Recall: 0.78\n",
      "F1 score: 0.81\n"
     ]
    }
   ],
   "source": [
    "train_lgbm_model(cars_df, test_size=0.2, max_depth=3, n_estimators=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "mFcmm9y107Ag"
   },
   "source": [
    "#### Experiment 3 - XGBoost model\n",
    "\n",
    "Let's train 3 models, changing the parameters max_depth and n_estimators."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "NSladWsI04jh"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating train and test data...\n",
      "Training model...\n",
      "Associating projects/1080686785400/locations/europe-west4/metadataStores/default/contexts/my-experiment-xgboost-2023-07-27-18-58-20-feb4e to Experiment: my-experiment\n",
      "Evaluating model...\n",
      "Accuracy: 1.0\n",
      "Precision: 1.0\n",
      "Recall: 1.0\n",
      "F1 score: 1.0\n"
     ]
    }
   ],
   "source": [
    "train_xgboost_model(cars_df, test_size=0.2, max_depth=3, n_estimators=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "zEfU4BHzyAM0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating train and test data...\n",
      "Training model...\n",
      "Associating projects/1080686785400/locations/europe-west4/metadataStores/default/contexts/my-experiment-xgboost-2023-07-27-18-58-28-36f71 to Experiment: my-experiment\n",
      "Evaluating model...\n",
      "Accuracy: 0.988\n",
      "Precision: 1.0\n",
      "Recall: 0.833\n",
      "F1 score: 0.909\n"
     ]
    }
   ],
   "source": [
    "train_xgboost_model(cars_df, test_size=0.2, max_depth=5, n_estimators=100)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "TgMhMXfTyAM0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Generating train and test data...\n",
      "Training model...\n",
      "Associating projects/1080686785400/locations/europe-west4/metadataStores/default/contexts/my-experiment-xgboost-2023-07-27-18-58-37-b317a to Experiment: my-experiment\n",
      "Evaluating model...\n",
      "Accuracy: 0.994\n",
      "Precision: 1.0\n",
      "Recall: 0.913\n",
      "F1 score: 0.955\n"
     ]
    }
   ],
   "source": [
    "train_xgboost_model(cars_df, test_size=0.2, max_depth=10, n_estimators=200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hAQUlys2j0UN"
   },
   "source": [
    "#### Compare experiment results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "2XcCQUCgdTmG"
   },
   "outputs": [],
   "source": [
    "experiment_df = vertex_ai.get_experiment_df()\n",
    "experiment_df.T"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "environment": {
   "kernel": "python3",
   "name": "tf2-gpu.2-11.m109",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-11:m109"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}