lessons/00-intro/practice-intro-subject.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use Case Overview: \n",
    "\n",
    "The objective of this notebook is to predict the duration of NYC taxi trips. \\\n",
    "We use data related to yellow or green taxis to train a simple prediction model.\n",
    "\n",
    "In this notebook, we use:\n",
    "\n",
    "- Yellow taxi data from January 2021 for model training.\n",
    "- Data from February 2021 to test the model (make predictions).\n",
    "\n",
    "**Features for Model Training**: Useful variables related to the trip itself \n",
    "\n",
    "**Target to Predict**: The duration of the trip\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data\n",
    "Data for the whole course can be downloaded following this [link](https://www.nyc.gov/site/tlc/about/tlc-trip-record-data.page) or using the following code:\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install gdown"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import gdown\n",
    "import os\n",
    "\n",
    "DATA_FOLDER = \"../../data\"\n",
    "train_path = f\"{DATA_FOLDER}/yellow_tripdata_2021-01.parquet\"\n",
    "test_path = f\"{DATA_FOLDER}/yellow_tripdata_2021-02.parquet\"\n",
    "predict_path = f\"{DATA_FOLDER}/yellow_tripdata_2021-03.parquet\"\n",
    "\n",
    "\n",
    "if not os.path.exists(DATA_FOLDER):\n",
    "    os.makedirs(DATA_FOLDER)\n",
    "    print(f\"New directory {DATA_FOLDER} created!\")\n",
    "\n",
    "gdown.download(\n",
    "    \"https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2021-01.parquet\",\n",
    "    train_path,\n",
    "    quiet=False,\n",
    ")\n",
    "gdown.download(\n",
    "    \"https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2021-02.parquet\",\n",
    "    test_path,\n",
    "    quiet=False,\n",
    ")\n",
    "gdown.download(\n",
    "    \"https://d37ci6vzurychx.cloudfront.net/trip-data/yellow_tripdata_2021-03.parquet\",\n",
    "    predict_path,\n",
    "    quiet=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from sklearn.feature_extraction import DictVectorizer\n",
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.metrics import mean_absolute_percentage_error, root_mean_squared_error\n",
    "import warnings\n",
    "\n",
    "warnings.filterwarnings(\"ignore\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Info** : <p style=color:green>Using large amounts of data in jupyter notebook, some cell can take some time to run<p/>"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1 - Load data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "DATA_FOLDER = \"../../data\"\n",
    "\n",
    "train_df = pd.read_parquet(os.path.join(DATA_FOLDER, \"yellow_tripdata_2021-01.parquet\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "train_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df.isnull().sum()"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2 - Prepare the data"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2-1 Compute the target"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset contains pickup and dropoff time but not the duration itself. \\\n",
    "We compute the duration of a taxi trip in minutes using these two variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_target(df):\n",
    "    df[\"duration\"] = df[\"tpep_dropoff_datetime\"] - df[\"tpep_pickup_datetime\"]\n",
    "    df[\"duration\"] = df[\"duration\"].dt.total_seconds() / 60\n",
    "    return df\n",
    "\n",
    "\n",
    "train_df = compute_target(train_df)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can visualise how the duration is distributed : "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "train_df[\"duration\"].describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ax = plt.figure(figsize=(12, 6))\n",
    "sns.displot(train_df.duration, ax=ax, kind=\"kde\")"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We notice that there are negative durations and trips that last 6 hours. \\\n",
    "We will proceed to remove outliers and narrow the scope to trips lasting between 1 minute and 1 hour."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "MIN_DURATION = 1\n",
    "MAX_DURATION = 60\n",
    "\n",
    "\n",
    "def filter_outliers(df, min_duration=MIN_DURATION, max_duration=MAX_DURATION):\n",
    "    df = df[df[\"duration\"].between(min_duration, max_duration)]\n",
    "    return df\n",
    "\n",
    "\n",
    "train_df = filter_outliers(train_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "ax = plt.figure(figsize=(12, 6))\n",
    "sns.displot(train_df.duration, ax=ax)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2-2 Prepare features"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2-2-1 Categorical features"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will encode discrete variables as strings and then proceed to extracting the features and the target in order to train two models:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "CATEGORICAL_COLS = [\"PULocationID\", \"DOLocationID\", \"passenger_count\"]\n",
    "\n",
    "\n",
    "def encode_categorical_cols(df):\n",
    "    df[CATEGORICAL_COLS] = df[CATEGORICAL_COLS].fillna(-1).astype(\"int\")\n",
    "    df[CATEGORICAL_COLS] = df[CATEGORICAL_COLS].astype(\"category\")\n",
    "    return df\n",
    "\n",
    "\n",
    "train_df = encode_categorical_cols(train_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def extract_x_y(df, dv=None):\n",
    "    dicts = df[CATEGORICAL_COLS].to_dict(orient=\"records\")\n",
    "    if dv is None:\n",
    "        dv = DictVectorizer()\n",
    "        dv.fit(dicts)\n",
    "    X = dv.transform(dicts)\n",
    "    y = df[\"duration\"].values\n",
    "    return X, y, dv\n",
    "\n",
    "\n",
    "X_train, y_train, dv = extract_x_y(train_df)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3 - Train model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = LinearRegression()\n",
    "lr.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ~ 2.5 minutes\n",
    "rf = RandomForestRegressor(n_estimators=100, max_depth=11, max_features=\"sqrt\", random_state=1, n_jobs=-1)\n",
    "rf.fit(X_train, y_train)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 4 - Evaluate model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_metrics(y, y_pred):\n",
    "    metrics = {\n",
    "        \"rmse\": root_mean_squared_error(y, y_pred),\n",
    "        \"mape\": mean_absolute_percentage_error(y, y_pred),\n",
    "    }\n",
    "    return metrics"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4-1 On train data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred_lr = lr.predict(X_train)\n",
    "y_pred_rf = rf.predict(X_train)\n",
    "compute_metrics(y_train, y_pred_lr), compute_metrics(y_train, y_pred_rf)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4-2 On test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df = pd.read_parquet(os.path.join(DATA_FOLDER, \"yellow_tripdata_2021-02.parquet\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_df = compute_target(test_df)\n",
    "test_df = filter_outliers(test_df)\n",
    "test_df = encode_categorical_cols(test_df)\n",
    "X_test, y_test, _ = extract_x_y(test_df, dv=dv)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred_test_lr = lr.predict(X_test)\n",
    "y_pred_test_rf = rf.predict(X_test)\n",
    "compute_metrics(y_test, y_pred_test_lr), compute_metrics(y_test, y_pred_test_rf)"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusion\n",
    "\n",
    "* What can you say of the trained models?\n",
    "* If you had to improve these models, what would you do? "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.15"
  },
  "vscode": {
   "interpreter": {
    "hash": "cf9f4cdb3edcfc4a863bd855df3eca2688bc43b4597668224922768b074a97a8"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}