v1/python-sdk/tutorials/automl-with-azureml/forecasting-data-preparation/auto-ml-forecasting-data-preparation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright (c) Microsoft Corporation. All rights reserved.\n",
    "\n",
    "Licensed under the MIT License."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Automated Machine Learning\n",
    "**Data Preparation for Demand Forecasting Notebooks**\n",
    "\n",
    "## Contents\n",
    "1. [Introduction](#Introduction)\n",
    "1. [Setup](#Setup)\n",
    "1. [Data Pre-processing](#DataWork)\n",
    "1. [Dealing with Duplicates](#DealWithDuplicates)\n",
    "1. [Data Partitioning](#DataPartition)\n",
    "1. [Data Upload](#DataUpload)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Introduction\n",
    "\n",
    "The objective of this notebook is to illustrate how to pre-process the raw data and register partitioned datasets to be used in demand forecasting notebooks:<br><ol><li> [Demand Forecastsing Using TCN](https://github.com/Azure/azureml-examples/blob/main/v1/python-sdk/tutorials/automl-with-azureml/forecasting-demand-forecasting-tcn/auto-ml-forecasting-demand-tcn.ipynb)</li><li> Demand Forecasting Using Many Models ([link placeholder]())</li></ol>\n",
    "For illustration purposes we use the UCI electricity data ([link](https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014#)) which contains electricity consumption data for 370 consumers measured at 15 minute intervals. In this notebook, we will show how to ingest the data from the original source, aggregate to an hourly frequency, select a subsample of unique time series, determine the approriate way to partition the data, and finally, register the datasets to be used on the aforementoned notebooks.\n",
    "\n",
    "Make sure you have executed the [configuration](https://github.com/Azure/MachineLearningNotebooks/blob/master/configuration.ipynb) before running this notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import logging\n",
    "import os\n",
    "import random\n",
    "\n",
    "import azureml.core\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "from azureml.core.workspace import Workspace\n",
    "from matplotlib import pyplot as plt\n",
    "\n",
    "random.seed(12345)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Accessing the Azure ML workspace requires authentication with Azure. The default authentication is an interactive authentication using the default tenant. Executing the `ws = Workspace.from_config()` line in the cell below will prompt for authentication the first time that it is run.\n",
    "\n",
    "If you have multiple Azure tenants, you can specify the tenant by replacing the ws = Workspace.from_config() line in the cell below with the following:\n",
    "```\n",
    "from azureml.core.authentication import InteractiveLoginAuthentication\n",
    "auth = InteractiveLoginAuthentication(tenant_id = 'mytenantid')\n",
    "ws = Workspace.from_config(auth = auth)\n",
    "```\n",
    "If you need to run in an environment where interactive login is not possible, you can use Service Principal authentication by replacing the ws = Workspace.from_config() line in the cell below with the following:\n",
    "```\n",
    "from azureml.core.authentication import ServicePrincipalAuthentication\n",
    "auth = ServicePrincipalAuthentication('mytenantid', 'myappid', 'mypassword')\n",
    "ws = Workspace.from_config(auth = auth)\n",
    "```\n",
    "For more details, see [aka.ms/aml-notebook-auth](aka.ms/aml-notebook-auth)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ws = Workspace.from_config()\n",
    "datastore = ws.get_default_datastore()\n",
    "\n",
    "output = {}\n",
    "output[\"Subscription ID\"] = ws.subscription_id\n",
    "output[\"Workspace\"] = ws.name\n",
    "output[\"Resource Group\"] = ws.resource_group\n",
    "output[\"Location\"] = ws.location\n",
    "pd.set_option(\"display.max_colwidth\", None)\n",
    "outputDf = pd.DataFrame(data=output, index=[\"\"])\n",
    "outputDf.T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Data work\n",
    "We start with declaring some of the parameters that will be used in this notebook.\n",
    "\n",
    "- `IS_ORIGINAL_DATASET_DOWNLOADED` is a flag for wether we want to download the original data from the source. The flag is here to reduce the download time since the original dataset is larger than 1 GB.\n",
    "- `IS_MODIFIED_DATASET_UPLOADED` is a flag for whether the datasets are uploaded to the Datastore. We use it to prevent unintentional uploads of the same datasets.\n",
    "- `DOES_PARTITION_INCLUDE_VALIDATION_SET` is a placeholder for determining whether the partitioned data should include the validation set. The value True/False will be determined later in the notebook."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "IS_ORIGINAL_DATASET_DOWNLOADED = False\n",
    "IS_MODIFIED_DATASET_UPLOADED = False\n",
    "DOES_PARTITION_INCLUDE_VALIDATION_SET = (\n",
    "    None  # place holder for the parameter value we will determine later.\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we specify parameters specific to the data we will work with.\n",
    "\n",
    "- **Target column** is what we want to forecast. In our case it is electricity consumption per customer measured in kilowatt hours (kWh).\n",
    "- **Time column** is the time axis along which to predict.\n",
    "- **Time series identifier columns** are identified by values of the columns listed `time_series_id_column_names`. In our case all unique time series are identified by a single column `customer_id`. However, it is quite common to have multiple columns identifying unique time series. See the link for a more detailed explanation on this topic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "target_column_name = \"usage\"\n",
    "time_column_name = \"date\"\n",
    "SERIES_ID = \"customer_id\"\n",
    "time_series_id_column_names = [SERIES_ID]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the followng block of code we will download the data from the original source to the `data` folder, load the data and print the first few rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not IS_ORIGINAL_DATASET_DOWNLOADED:\n",
    "    print(\"Downloading data from the source ...\\n---\")\n",
    "    # Download original data\n",
    "    from io import BytesIO\n",
    "    from urllib.request import urlopen\n",
    "    from zipfile import ZipFile\n",
    "\n",
    "    zipurl = \"https://archive.ics.uci.edu/ml/machine-learning-databases/00321/LD2011_2014.txt.zip\"\n",
    "\n",
    "    with urlopen(zipurl) as zipresp:\n",
    "        with ZipFile(BytesIO(zipresp.read())) as zfile:\n",
    "            zfile.extractall(\"data\")\n",
    "    IS_ORIGINAL_DATASET_DOWNLOADED = True\n",
    "\n",
    "DATA_LOCATION = os.path.join(os.getcwd(), \"data\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Printing the first few rows of the downloaded data ...\\n---\")\n",
    "raw_df = pd.read_table(\"data/LD2011_2014.txt\", sep=\";\", low_memory=False)\n",
    "print(raw_df.head())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_df.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The downloaded data is in \"wide\" format, meaning each column name that starts with \"MT_xxx\" represents one unique time series. The first columnn \"Unnamed: 0\" is actually a time stamp at which the obervation for every time series takes place. Let's rename this column to something more meaningful. We will call it `date`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_df.rename(columns={raw_df.columns[0]: \"date\"}, inplace=True)\n",
    "raw_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The \"wide\" data format is not usable in AutoML, which is designed to accept a \"long\" format data, the same format that is accepted by typical scikit-learn machine learning models. To tranform the data from wide to long format, we will take each uniqe time series (date, MT_xxx) and stack them vertically. The end result will be a data frame containing 3 columns: (i) the `date` column, (ii) `custmer_id` column, which is the identifier of the time series and is derived from the column name in the wide format, and (iii) `usage` which is the target varaible we are trying to model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Converting data from wide to long format. This may take a few minutes ...\\n---\")\n",
    "raw_df = pd.melt(raw_df, id_vars=\"date\", var_name=\"customer_id\", value_name=\"usage\")\n",
    "raw_df[time_column_name] = pd.to_datetime(raw_df[time_column_name])\n",
    "raw_df.to_csv(\"data/LD2011_2014_long_format.csv\", index=False)\n",
    "\n",
    "print(\"The first few rows of the data in the long format ...\\n---\")\n",
    "raw_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nseries = raw_df.groupby(time_series_id_column_names).ngroups\n",
    "print(\"Data contains {0} individual time-series.\".format(nseries))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data tracks customers' electricity consumption every 15 minutes and is measured in kilowatt (kW) consumed. Let's assume the business requirement is to generate 24 hour forecast in kilowatt hours (kWh). Such forecast at 15 minute frequency results in the forecast horizon of 96 steps ahead (there are 96 15-minute intervals in a 24-hour period). Moreover, if the requirement is to generate 24-hour ahead forecast, it makes more sence to aggregate data measured at 15-minute intervals to an hourly frequency. This will reduce the forecast horizon by a factor of 4. The shorter the forecast horizon usually results in higher probability of achieving better forecast accuracy. \n",
    "\n",
    "In the next block of code we will create a `datetime` column which will identify the date and the hour of the day each observation belogs to. We also convert the target variable from kW to kWh, where $kWh = \\frac{1}{4} \\times kW $. After the conversion is complete, the hourly data will be stored in the `raw_hourly_df` object."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of the previous command shows that the target varaible `usage` is an object. We need to transform it into a float in order to convert kW to kWh. The next command does exactly this. Because the original data contains European style format with decimals separated by commmas, we replace commas with periods before declaring the target variaible as a float."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_df[target_column_name] = (\n",
    "    raw_df[target_column_name].astype(str).apply(lambda x: float(x.replace(\",\", \".\")))\n",
    ")\n",
    "raw_df.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# aggregate data to hourly. Here, the hourly column is called \"datetime\"\n",
    "new_time_column_name = \"datetime\"\n",
    "\n",
    "raw_df[new_time_column_name] = raw_df[time_column_name].dt.to_period(\"H\")\n",
    "raw_df[target_column_name] = raw_df[target_column_name] / 4  # convert to kWh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert to hourly consumption by adding kWh for every 15 min interval\n",
    "raw_hourly_series = raw_df.groupby([SERIES_ID, new_time_column_name])[\n",
    "    target_column_name\n",
    "].sum()\n",
    "raw_hourly_df = pd.DataFrame(raw_hourly_series)\n",
    "raw_hourly_df.reset_index(drop=False, inplace=True)\n",
    "print(raw_hourly_df.head())\n",
    "\n",
    "del raw_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert time column to the datetime format\n",
    "raw_hourly_df[new_time_column_name] = pd.to_datetime(\n",
    "    raw_hourly_df[new_time_column_name].astype(str)\n",
    ")\n",
    "raw_hourly_df.to_csv(\n",
    "    os.path.join(DATA_LOCATION, \"hourly_data_long_format.csv\"), index=False\n",
    ")\n",
    "raw_hourly_df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, let's visualize a sample of 50 randomly selected series. The plots will be stored in the `output_folder`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_grains = list(\n",
    "    pd.unique(raw_hourly_df[time_series_id_column_names].values.ravel(\"K\"))\n",
    ")\n",
    "grains_to_plot = random.sample(all_grains, k=50)\n",
    "print(f\"The following grains will be selected for plotting: \\n{grains_to_plot}\\n---\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_subset = raw_hourly_df[raw_hourly_df[SERIES_ID].isin(grains_to_plot)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create an output folder\n",
    "OUTPUT_DIR = os.path.join(os.getcwd(), \"output_folder\")\n",
    "os.makedirs(OUTPUT_DIR, exist_ok=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from scripts.helper_scripts import _draw_one_plot\n",
    "from matplotlib.backends.backend_pdf import PdfPages\n",
    "\n",
    "plot_filename = \"raw_ts_plots.pdf\"\n",
    "\n",
    "pdf = PdfPages(os.path.join(OUTPUT_DIR, plot_filename))\n",
    "for grain, one_forecast in data_subset.groupby(SERIES_ID):\n",
    "    one_forecast[new_time_column_name] = pd.to_datetime(\n",
    "        one_forecast[new_time_column_name]\n",
    "    )\n",
    "    one_forecast.sort_values(new_time_column_name, inplace=True)\n",
    "    _draw_one_plot(\n",
    "        one_forecast,\n",
    "        new_time_column_name,\n",
    "        target_column_name,\n",
    "        time_series_id_column_names,\n",
    "        pdf,\n",
    "        plot_predictions=False,\n",
    "    )\n",
    "pdf.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import IFrame\n",
    "\n",
    "IFrame(\"./output_folder/raw_ts_plots.pdf\", width=800, height=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Close examination of the consumption plots per customer shows there are quite a few customers that have no usage data prior to January 1, 2012. Some customers do not have the data until January of 2013 or 2014. It is reasonable at this point to drop all observations prior to January 1, 2012. We will call this object `clean_df`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop grains that have no usage as of Jan 1, 2012\n",
    "tmp_df = raw_hourly_df[raw_hourly_df[new_time_column_name] == \"2012-01-01 01:00:00\"]\n",
    "grains_to_drop = list(tmp_df[tmp_df[target_column_name] == 0][SERIES_ID])\n",
    "print(f\"Number of grains to be dropped: {len(grains_to_drop)}\")\n",
    "del tmp_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "clean_df = raw_hourly_df[~raw_hourly_df[SERIES_ID].isin(grains_to_drop)]\n",
    "\n",
    "# drop observations prior to 1/1/2012 since they are zero for all grains\n",
    "clean_df = clean_df[clean_df[new_time_column_name] > \"2011-12-31 23:00:00\"]\n",
    "\n",
    "del raw_hourly_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To save training runtime, we will use a small subset of 10 unique time series from the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_grains = list(pd.unique(clean_df[time_series_id_column_names].values.ravel(\"K\")))\n",
    "selected_grains = random.sample(all_grains, k=10)\n",
    "print(f\"The following grains will be selected:  {selected_grains}\\n---\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_subset = clean_df[clean_df[SERIES_ID].isin(selected_grains)]\n",
    "nseries = data_subset.groupby(time_series_id_column_names).ngroups\n",
    "print(\"Data subset contains {0} individual time-series.\\n---\".format(nseries))\n",
    "data_subset.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "del all_grains, selected_grains, clean_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Full dataset\\n---\")\n",
    "for grain, tmp_df in data_subset.groupby(time_series_id_column_names):\n",
    "    print(\n",
    "        \"Grain:{}.\\\n",
    "    Min date: {}\\\n",
    "    Max date: {}\\\n",
    "    N: {}\".format(\n",
    "            grain,\n",
    "            tmp_df[new_time_column_name].min(),\n",
    "            tmp_df[new_time_column_name].max(),\n",
    "            tmp_df.shape[0],\n",
    "        )\n",
    "    )\n",
    "del tmp_df, grain"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_subset.to_csv(os.path.join(OUTPUT_DIR, \"small_data.csv\"), index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Dealing with Duplicates\n",
    "\n",
    "In this section we will check for duplicate values in the data, i.e., there are several observations associated with the same time stamp for at least one unique time series. For example, if duplicate values were present in our data, it would look like the following:\n",
    "\n",
    "| customer_id | datetime         | usage |\n",
    "|:--          | :--              |:--:   |\n",
    "| ...         | ...              | ...   |\n",
    "|MT_001       | 2012-01-01 15:00 | ...   |\n",
    "|MT_001       | 2012-01-01 15:00 | ...   |\n",
    "| ...         | ...              | ...   |\n",
    "\n",
    "In this example, there 2 observations associated with January 1, 2012 3:00 PM time stamp for the customer ID `MT_001`.  AutoML will throw a user error if such scenario was encountered because it does not which value to use. The following block of code checks for a total number of duplicates in the data as well as as give us the breakdown per time series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "duplicate_observations = data_subset.duplicated(\n",
    "    subset=[new_time_column_name, SERIES_ID], keep=False\n",
    ").sum()\n",
    "print(\n",
    "    f\"---\\nTotal duplicates: {data_subset.duplicated(subset=[new_time_column_name, SERIES_ID], keep=False).sum()}\\n---\"\n",
    ")\n",
    "for grain, tmp_df in data_subset.groupby(SERIES_ID):\n",
    "    print(\n",
    "        f\"Zone: {grain}. Number of duplicates: {tmp_df.duplicated(subset=[new_time_column_name, SERIES_ID], keep=False).sum()}\"\n",
    "    )\n",
    "del tmp_df, grain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we remove duplicates from the data if they are present."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if duplicate_observations > 0:\n",
    "    print(\n",
    "        f\"Removing duplicate observations\\n---\\nOriginal size: {data_subset.shape}\\n---\"\n",
    "    )\n",
    "    data_subset.drop_duplicates(\n",
    "        subset=[new_time_column_name, SERIES_ID], ignore_index=True, inplace=True\n",
    "    )\n",
    "    print(f\"Cleaned size: {data_subset.shape}\\n---\")\n",
    "\n",
    "    for grain, tmp_df in data_subset.groupby(SERIES_ID):\n",
    "        print(\n",
    "            f\"Zone: {grain}. Number of duplicates: {tmp_df.duplicated(subset=[new_time_column_name], keep=False).sum()}\"\n",
    "        )\n",
    "    del tmp_df, grain"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Data Partitioning\n",
    "The objective of this section is to determine whether you want to use the [many models solution accelerator](https://github.com/Azure/azureml-examples/tree/main/v1/python-sdk/tutorials/automl-with-azureml/forecasting-many-models) or a [deep learning model](https://learn.microsoft.com/en-us/azure/machine-learning/concept-automl-forecasting-deep-learning). Many models approach allows users to train and manage models for millions of time series in parallel and may be an appropriate modelling choice when time series in your dataset exhibit heterogeneous behavior. During the model selection stage AutoML searches for the best non-DNN model or a combination of models (ensemble) for each time series or a group of time series. Please refer to this [link](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-forecast#many-models) for the workflow diagram for such framework. When using many models solution accelerator you do not need a validation set because AutoML uses a [rolling origin cross validation](https://learn.microsoft.com/en-us/azure/machine-learning/how-to-auto-train-forecast#many-models) on the training data for model selection. As a result, the data will need to be partitioned into train, test and inference sets.\n",
    "\n",
    "The deep learning approach allows us to train one model for all time series in the dataset because it can learn complex patterns. On average, the TCNForcaster requires a less frequent re-training compared to the many-models approach. Because of this, the user is expected to provide a validation set which is used to search for the best architecture.Thus, the data will need to be partitioned into train, test, validation and inference sets.\n",
    "\n",
    "The difference between the *test* and *inference* sets is the presence of the target column. The test set contains the target column and will be used to evaluate model performance using [rolling forecast](https://learn.microsoft.com/en-us/azure/machine-learning/v1/how-to-auto-train-forecast-v1#evaluating-model-accuracy-with-a-rolling-forecast). On the other hand, the target column is not present in the inference set to illustrate how to generate an actual forecast.\n",
    "\n",
    "Before making this decision, let's visualize the small subset of data we selected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_filename = \"ts_plots_small_data.pdf\"\n",
    "\n",
    "pdf = PdfPages(os.path.join(OUTPUT_DIR, plot_filename))\n",
    "for grain, one_forecast in data_subset.groupby(SERIES_ID):\n",
    "    one_forecast[new_time_column_name] = pd.to_datetime(\n",
    "        one_forecast[new_time_column_name]\n",
    "    )\n",
    "    one_forecast.sort_values(new_time_column_name, inplace=True)\n",
    "    _draw_one_plot(\n",
    "        one_forecast,\n",
    "        new_time_column_name,\n",
    "        target_column_name,\n",
    "        time_series_id_column_names,\n",
    "        pdf,\n",
    "        plot_predictions=False,\n",
    "    )\n",
    "pdf.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from IPython.display import IFrame\n",
    "\n",
    "IFrame(\"./output_folder/ts_plots_small_data.pdf\", width=800, height=300)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One way to determine the modelling framework is by performing a visual examination of the raw time series plots. If all time series exhibit similar behavior patterns, a deep learning model can be an excellent choice. If, on the other hand, individual time series show heterogeneous behavior, it is advised to run a many models accelerator which estimates one model per time seires as oppsed to one model for all time series.\n",
    "\n",
    "In our case, it seems like a deep learning model could be a good modeeling choice since the time series look fairly similar. As a result, we set the `DOES_PARTITION_INCLUDE_VALIDATION_SET` parameter to True. Please note that to explore the best option, you can still partition the data to run the low capacity models. To do so, set this parametr to False."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DOES_PARTITION_INCLUDE_VALIDATION_SET = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Generate train/valid/test/inference sets\n",
    "\n",
    "Since deep learning models are considered \"high capacity\" models, they generally do not require frequent re-training. As a result, we use 2 months of data for validation and test sets, respectively. The choice of 2 months is fairly arbitrary and can be modified to suit your needs. We use 2 months of validation and test sets data to reflect the infrequent re-traiing of the model given that the data frequency is hourly. Thus, there will be more than 1200 observations in the vlaidation and test sets per time series. This will give us enough data points to generate conclusions about the model's performance.\n",
    "\n",
    "This is in constrast to the ML models that require frequent retraining and, as a result, require much shorter test sets to have a reasonable understanding of the model accuracy.\n",
    "\n",
    "**Note:** Once the backtesting functionality is available, replace the statement regarding the shorter test set with the need of backtesting given a relatively frequent need to re-train the models compared to the DNNs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "n_test_periods = 60 * 24\n",
    "n_valid_periods = 60 * 24  # applicable only to single/TCN model\n",
    "n_inference_periods = 24\n",
    "\n",
    "\n",
    "def split_last_n_by_series_id(df, n, time_column_name):\n",
    "    \"\"\"Group df by series identifiers and split on last n rows for each group.\"\"\"\n",
    "    df_grouped = df.sort_values(time_column_name).groupby(  # Sort by ascending time\n",
    "        time_series_id_column_names, group_keys=False\n",
    "    )\n",
    "    df_head = df_grouped.apply(lambda dfg: dfg.iloc[:-n])\n",
    "    df_tail = df_grouped.apply(lambda dfg: dfg.iloc[-n:])\n",
    "    return df_head, df_tail\n",
    "\n",
    "\n",
    "train_valid_test, inference = split_last_n_by_series_id(\n",
    "    data_subset, n_inference_periods, time_column_name=new_time_column_name\n",
    ")\n",
    "\n",
    "if DOES_PARTITION_INCLUDE_VALIDATION_SET:\n",
    "    train_valid, test = split_last_n_by_series_id(\n",
    "        train_valid_test, n_test_periods, new_time_column_name\n",
    "    )\n",
    "    train, valid = split_last_n_by_series_id(\n",
    "        train_valid, n_valid_periods, new_time_column_name\n",
    "    )\n",
    "else:\n",
    "    train, test = split_last_n_by_series_id(\n",
    "        train_valid_test, n_test_periods, new_time_column_name\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We drop the target column from the inference dataset to reflect the fact that the future is unknown and the forecast is our best guess about it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "inference.drop(columns=[target_column_name], inplace=True)\n",
    "inference.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we will examine the start and end dates as well as the number of observations per time series in each of the generated datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Full dataset\\n---\")\n",
    "for grain, tmp_df in data_subset.groupby(time_series_id_column_names):\n",
    "    print(\n",
    "        \"Grain:{}.\\\n",
    "    Min date: {}\\\n",
    "    Max date: {}\\\n",
    "    N: {}\".format(\n",
    "            grain, tmp_df[\"datetime\"].min(), tmp_df[\"datetime\"].max(), tmp_df.shape[0]\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Train dataset\\n---\")\n",
    "for grain, tmp_df in train.groupby(time_series_id_column_names):\n",
    "    print(\n",
    "        \"Grain:{}.\\\n",
    "    Min date: {}\\\n",
    "    Max date: {}\\\n",
    "    N: {}\".format(\n",
    "            grain, tmp_df[\"datetime\"].min(), tmp_df[\"datetime\"].max(), tmp_df.shape[0]\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if DOES_PARTITION_INCLUDE_VALIDATION_SET:\n",
    "    print(\"Valid dataset\\n---\")\n",
    "    for grain, tmp_df in valid.groupby(time_series_id_column_names):\n",
    "        print(\n",
    "            \"Grain:{}.\\\n",
    "        Min date: {}\\\n",
    "        Max date: {}\\\n",
    "        N: {}\".format(\n",
    "                grain,\n",
    "                tmp_df[\"datetime\"].min(),\n",
    "                tmp_df[\"datetime\"].max(),\n",
    "                tmp_df.shape[0],\n",
    "            )\n",
    "        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(\"Test dataset\\n---\")\n",
    "for grain, tmp_df in test.groupby(time_series_id_column_names):\n",
    "    print(\n",
    "        \"Grain:{}.\\\n",
    "    Min date: {}\\\n",
    "    Max date: {}\\\n",
    "    N: {}\".format(\n",
    "            grain, tmp_df[\"datetime\"].min(), tmp_df[\"datetime\"].max(), tmp_df.shape[0]\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Inference dataset\\n---\")\n",
    "for grain, tmp_df in inference.groupby(time_series_id_column_names):\n",
    "    print(\n",
    "        \"Grain:{}.\\\n",
    "    Min date: {}\\\n",
    "    Max date: {}\\\n",
    "    N: {}\".format(\n",
    "            grain, tmp_df[\"datetime\"].min(), tmp_df[\"datetime\"].max(), tmp_df.shape[0]\n",
    "        )\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 6.  Upload data to datastore\n",
    "The [Machine Learning service workspace](https://docs.microsoft.com/en-us/azure/machine-learning/service/concept-workspace), is paired with an storage account, which contains the default data store. We will use it to upload the train and test sets data and create [tabular datasets](https://docs.microsoft.com/en-us/python/api/azureml-core/azureml.data.tabulardataset?view=azure-ml-py) for training and testing. A tabular dataset defines a series of lazily-evaluated, immutable operations to load data from the data source into tabular representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATASET_PREFFIX_NAME = (\n",
    "    \"uci_electro_small_tcn\"\n",
    "    if DOES_PARTITION_INCLUDE_VALIDATION_SET\n",
    "    else \"uci_electro_small\"\n",
    ")\n",
    "print(f\"Dataset preffix name: {DATASET_PREFFIX_NAME}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "IS_MODIFIED_DATASET_UPLOADED"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from azureml.data.dataset_factory import TabularDatasetFactory\n",
    "from azureml.core.dataset import Dataset\n",
    "\n",
    "if not IS_MODIFIED_DATASET_UPLOADED:\n",
    "    print(\"---\\nUploading data ...\\n---\")\n",
    "\n",
    "    train_dataset = TabularDatasetFactory.register_pandas_dataframe(\n",
    "        train, target=(datastore, \"dataset/\"), name=f\"{DATASET_PREFFIX_NAME}_train\"\n",
    "    )\n",
    "\n",
    "    if DOES_PARTITION_INCLUDE_VALIDATION_SET:\n",
    "        valid_dataset = TabularDatasetFactory.register_pandas_dataframe(\n",
    "            valid, target=(datastore, \"dataset/\"), name=f\"{DATASET_PREFFIX_NAME}_valid\"\n",
    "        )\n",
    "\n",
    "    test_dataset = TabularDatasetFactory.register_pandas_dataframe(\n",
    "        test, target=(datastore, \"dataset/\"), name=f\"{DATASET_PREFFIX_NAME}_test\"\n",
    "    )\n",
    "\n",
    "    inference_dataset = TabularDatasetFactory.register_pandas_dataframe(\n",
    "        inference,\n",
    "        target=(datastore, \"dataset/\"),\n",
    "        name=f\"{DATASET_PREFFIX_NAME}_inference\",\n",
    "    )\n",
    "else:\n",
    "    print(\"Using uploaded data ...\\n---\")\n",
    "\n",
    "    target_path_train = f\"{DATASET_PREFFIX_NAME}_train\"\n",
    "    target_path_valid = f\"{DATASET_PREFFIX_NAME}_valid\"\n",
    "    target_path_test = f\"{DATASET_PREFFIX_NAME}_test\"\n",
    "    target_path_inference = f\"{DATASET_PREFFIX_NAME}_test\"\n",
    "\n",
    "    train_dataset = Dataset.get_by_name(ws, name=target_path_train)\n",
    "    if DOES_PARTITION_INCLUDE_VALIDATION_SET:\n",
    "        valid_dataset = Dataset.get_by_name(ws, name=target_path_valid)\n",
    "    test_dataset = Dataset.get_by_name(ws, name=target_path_test)\n",
    "    inference_dataset = Dataset.get_by_name(ws, name=target_path_inference)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# delete downladed data files to save space\n",
    "import shutil\n",
    "\n",
    "shutil.rmtree(\"data\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "jialiu"
   }
  ],
  "category": "tutorial",
  "celltoolbar": "Raw Cell Format",
  "compute": [
   "Remote"
  ],
  "datasets": [
   "Orange Juice Sales"
  ],
  "deployment": [
   "Azure Container Instance"
  ],
  "exclude_from_index": false,
  "framework": [
   "Azure ML AutoML"
  ],
  "friendly_name": "Forecasting orange juice sales with deployment",
  "index_order": 1,
  "kernelspec": {
   "display_name": "Python 3.8 - AzureML",
   "language": "python",
   "name": "python38-azureml"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "tags": [
   "None"
  ],
  "task": "Forecasting",
  "vscode": {
   "interpreter": {
    "hash": "6bd77c88278e012ef31757c15997a7bea8c943977c43d6909403c00ae11d43ca"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}