laptop.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# High Performance Jupyter\n",
    "\n",
    "## Good ol' PyData\n",
    "\n",
    "|<img src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/e/ed/Pandas_logo.svg/1200px-Pandas_logo.svg.png\" width=\"200\" /> | <img src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/3/31/NumPy_logo_2020.svg/2880px-NumPy_logo_2020.svg.png\" width=\"200\" /> | <img src=\"https://upload.wikimedia.org/wikipedia/commons/thumb/0/05/Scikit_learn_logo_small.svg/1280px-Scikit_learn_logo_small.svg.png\" width=\"200\" />|\n",
    "| -- | -- | -- |\n",
    "\n",
    "Let's do our usual analysis on a laptop-sized machine with a dataset that fits comfortably in memory. This notebook should execute on any machine with >4GB RAM.\n",
    "\n",
    "Outputs here are from a 2019 Macbook Pro (6 cores, 32GB RAM)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import datetime\n",
    "import s3fs\n",
    "import warnings\n",
    "warnings.simplefilter(\"ignore\")\n",
    "\n",
    "data_path = 's3://nyc-tlc/trip data'\n",
    "seed = 42"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Load and explore data\n",
    "\n",
    "We are using the [NYC Taxi data](https://www1.nyc.gov/site/tlc/about/tlc-trip-record-data.page), which contains several publicly-available datasets about taxi and rideshare rides taken in New York City.\n",
    "\n",
    "Data is available from 2009 to 2020, but for this exercise we will use 2019 data only. Take care when using other data, as the schemas in the CSV files changed over the years. Most notably, in mid-2016 latitude and longitude fields were replaced with more generic taxi zones for privacy reasons.\n",
    "\n",
    "Files are hosted in this S3 location: `s3://nyc-tlc/trip data`. The dataframe libraries we are using (`pandas`, `dask.dataframe`, `cudf`) all support reading directly from S3 so we don't have to download any files to our local filesystem.\n",
    "\n",
    "We can use the `s3fs` package to explore the files that are on S3. We are dealing with a public bucket so there is no need to authenticate (`anon=True`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(138,\n",
       " ['nyc-tlc/trip data/yellow_tripdata_2009-01.csv',\n",
       "  'nyc-tlc/trip data/yellow_tripdata_2009-02.csv',\n",
       "  'nyc-tlc/trip data/yellow_tripdata_2009-03.csv',\n",
       "  'nyc-tlc/trip data/yellow_tripdata_2009-04.csv',\n",
       "  'nyc-tlc/trip data/yellow_tripdata_2009-05.csv'],\n",
       " ['nyc-tlc/trip data/yellow_tripdata_2020-02.csv',\n",
       "  'nyc-tlc/trip data/yellow_tripdata_2020-03.csv',\n",
       "  'nyc-tlc/trip data/yellow_tripdata_2020-04.csv',\n",
       "  'nyc-tlc/trip data/yellow_tripdata_2020-05.csv',\n",
       "  'nyc-tlc/trip data/yellow_tripdata_2020-06.csv'])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fs = s3fs.S3FileSystem(anon=True)\n",
    "\n",
    "files = fs.glob('s3://nyc-tlc/trip data/yellow_tripdata_*')\n",
    "len(files), files[:5], files[-5:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "It looks like there's one file per month. Let's see how big the files are for 2019."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2019 avg size (MB): 650.0\n",
      "2019 total size (GB): 8.0\n"
     ]
    }
   ],
   "source": [
    "files_2019 = fs.glob('s3://nyc-tlc/trip data/yellow_tripdata_2019-*.csv')\n",
    "file_sizes_2019 = [fs.du(f) for f in files_2019] \n",
    "\n",
    "print(f'2019 avg size (MB): {np.round(np.mean(file_sizes_2019) / 1e6)}')\n",
    "print(f'2019 total size (GB): {np.round(np.sum(file_sizes_2019) / 1e9)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>The following cell pulls the file down from S3 and loads the full contents into a dataframe - it'll take a couple minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 22.4 s, sys: 2.23 s, total: 24.6 s\n",
      "Wall time: 1min 20s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "taxi = pd.read_csv(\n",
    "        fs.open(f'{data_path}/yellow_tripdata_2019-01.csv'),\n",
    "        parse_dates=['tpep_pickup_datetime', 'tpep_dropoff_datetime'],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Row count: 7667792\n",
      "Size in GB: 1.487551776\n"
     ]
    }
   ],
   "source": [
    "print(f\"Row count: {len(taxi)}\")\n",
    "print(f\"Size in GB: {taxi.memory_usage(deep=True).sum() / 1e9}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>VendorID</th>\n",
       "      <th>tpep_pickup_datetime</th>\n",
       "      <th>tpep_dropoff_datetime</th>\n",
       "      <th>passenger_count</th>\n",
       "      <th>trip_distance</th>\n",
       "      <th>RatecodeID</th>\n",
       "      <th>store_and_fwd_flag</th>\n",
       "      <th>PULocationID</th>\n",
       "      <th>DOLocationID</th>\n",
       "      <th>payment_type</th>\n",
       "      <th>fare_amount</th>\n",
       "      <th>extra</th>\n",
       "      <th>mta_tax</th>\n",
       "      <th>tip_amount</th>\n",
       "      <th>tolls_amount</th>\n",
       "      <th>improvement_surcharge</th>\n",
       "      <th>total_amount</th>\n",
       "      <th>congestion_surcharge</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>2019-01-01 00:46:40</td>\n",
       "      <td>2019-01-01 00:53:20</td>\n",
       "      <td>1</td>\n",
       "      <td>1.5</td>\n",
       "      <td>1</td>\n",
       "      <td>N</td>\n",
       "      <td>151</td>\n",
       "      <td>239</td>\n",
       "      <td>1</td>\n",
       "      <td>7.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1.65</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>9.95</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>2019-01-01 00:59:47</td>\n",
       "      <td>2019-01-01 01:18:59</td>\n",
       "      <td>1</td>\n",
       "      <td>2.6</td>\n",
       "      <td>1</td>\n",
       "      <td>N</td>\n",
       "      <td>239</td>\n",
       "      <td>246</td>\n",
       "      <td>1</td>\n",
       "      <td>14.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>16.30</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2</td>\n",
       "      <td>2018-12-21 13:48:30</td>\n",
       "      <td>2018-12-21 13:52:40</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>N</td>\n",
       "      <td>236</td>\n",
       "      <td>236</td>\n",
       "      <td>1</td>\n",
       "      <td>4.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>5.80</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2</td>\n",
       "      <td>2018-11-28 15:52:25</td>\n",
       "      <td>2018-11-28 15:55:45</td>\n",
       "      <td>5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "      <td>N</td>\n",
       "      <td>193</td>\n",
       "      <td>193</td>\n",
       "      <td>2</td>\n",
       "      <td>3.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>7.55</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2</td>\n",
       "      <td>2018-11-28 15:56:57</td>\n",
       "      <td>2018-11-28 15:58:33</td>\n",
       "      <td>5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2</td>\n",
       "      <td>N</td>\n",
       "      <td>193</td>\n",
       "      <td>193</td>\n",
       "      <td>2</td>\n",
       "      <td>52.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.3</td>\n",
       "      <td>55.55</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   VendorID tpep_pickup_datetime tpep_dropoff_datetime  passenger_count  \\\n",
       "0         1  2019-01-01 00:46:40   2019-01-01 00:53:20                1   \n",
       "1         1  2019-01-01 00:59:47   2019-01-01 01:18:59                1   \n",
       "2         2  2018-12-21 13:48:30   2018-12-21 13:52:40                3   \n",
       "3         2  2018-11-28 15:52:25   2018-11-28 15:55:45                5   \n",
       "4         2  2018-11-28 15:56:57   2018-11-28 15:58:33                5   \n",
       "\n",
       "   trip_distance  RatecodeID store_and_fwd_flag  PULocationID  DOLocationID  \\\n",
       "0            1.5           1                  N           151           239   \n",
       "1            2.6           1                  N           239           246   \n",
       "2            0.0           1                  N           236           236   \n",
       "3            0.0           1                  N           193           193   \n",
       "4            0.0           2                  N           193           193   \n",
       "\n",
       "   payment_type  fare_amount  extra  mta_tax  tip_amount  tolls_amount  \\\n",
       "0             1          7.0    0.5      0.5        1.65           0.0   \n",
       "1             1         14.0    0.5      0.5        1.00           0.0   \n",
       "2             1          4.5    0.5      0.5        0.00           0.0   \n",
       "3             2          3.5    0.5      0.5        0.00           0.0   \n",
       "4             2         52.0    0.0      0.5        0.00           0.0   \n",
       "\n",
       "   improvement_surcharge  total_amount  congestion_surcharge  \n",
       "0                    0.3          9.95                   NaN  \n",
       "1                    0.3         16.30                   NaN  \n",
       "2                    0.3          5.80                   NaN  \n",
       "3                    0.3          7.55                   NaN  \n",
       "4                    0.3         55.55                   NaN  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "taxi.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "VendorID                          int64\n",
       "tpep_pickup_datetime     datetime64[ns]\n",
       "tpep_dropoff_datetime    datetime64[ns]\n",
       "passenger_count                   int64\n",
       "trip_distance                   float64\n",
       "RatecodeID                        int64\n",
       "store_and_fwd_flag               object\n",
       "PULocationID                      int64\n",
       "DOLocationID                      int64\n",
       "payment_type                      int64\n",
       "fare_amount                     float64\n",
       "extra                           float64\n",
       "mta_tax                         float64\n",
       "tip_amount                      float64\n",
       "tolls_amount                    float64\n",
       "improvement_surcharge           float64\n",
       "total_amount                    float64\n",
       "congestion_surcharge            float64\n",
       "dtype: object"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "taxi.dtypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3.73 s, sys: 590 ms, total: 4.32 s\n",
      "Wall time: 4.33 s\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>VendorID</th>\n",
       "      <td>7667792.0</td>\n",
       "      <td>1.637</td>\n",
       "      <td>0.540</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>2.00</td>\n",
       "      <td>2.00</td>\n",
       "      <td>4.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>passenger_count</th>\n",
       "      <td>7667792.0</td>\n",
       "      <td>1.567</td>\n",
       "      <td>1.224</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>2.00</td>\n",
       "      <td>9.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>trip_distance</th>\n",
       "      <td>7667792.0</td>\n",
       "      <td>2.801</td>\n",
       "      <td>3.738</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.90</td>\n",
       "      <td>1.53</td>\n",
       "      <td>2.80</td>\n",
       "      <td>831.80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RatecodeID</th>\n",
       "      <td>7667792.0</td>\n",
       "      <td>1.058</td>\n",
       "      <td>0.678</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>99.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>PULocationID</th>\n",
       "      <td>7667792.0</td>\n",
       "      <td>165.501</td>\n",
       "      <td>66.392</td>\n",
       "      <td>1.0</td>\n",
       "      <td>130.00</td>\n",
       "      <td>162.00</td>\n",
       "      <td>234.00</td>\n",
       "      <td>265.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DOLocationID</th>\n",
       "      <td>7667792.0</td>\n",
       "      <td>163.753</td>\n",
       "      <td>70.364</td>\n",
       "      <td>1.0</td>\n",
       "      <td>113.00</td>\n",
       "      <td>162.00</td>\n",
       "      <td>234.00</td>\n",
       "      <td>265.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>payment_type</th>\n",
       "      <td>7667792.0</td>\n",
       "      <td>1.292</td>\n",
       "      <td>0.473</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.00</td>\n",
       "      <td>2.00</td>\n",
       "      <td>4.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fare_amount</th>\n",
       "      <td>7667792.0</td>\n",
       "      <td>12.409</td>\n",
       "      <td>262.072</td>\n",
       "      <td>-362.0</td>\n",
       "      <td>6.00</td>\n",
       "      <td>8.50</td>\n",
       "      <td>13.50</td>\n",
       "      <td>623259.86</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>extra</th>\n",
       "      <td>7667792.0</td>\n",
       "      <td>0.328</td>\n",
       "      <td>0.507</td>\n",
       "      <td>-60.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.50</td>\n",
       "      <td>535.38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mta_tax</th>\n",
       "      <td>7667792.0</td>\n",
       "      <td>0.497</td>\n",
       "      <td>0.053</td>\n",
       "      <td>-0.5</td>\n",
       "      <td>0.50</td>\n",
       "      <td>0.50</td>\n",
       "      <td>0.50</td>\n",
       "      <td>60.80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tip_amount</th>\n",
       "      <td>7667792.0</td>\n",
       "      <td>1.827</td>\n",
       "      <td>2.501</td>\n",
       "      <td>-63.5</td>\n",
       "      <td>0.00</td>\n",
       "      <td>1.43</td>\n",
       "      <td>2.33</td>\n",
       "      <td>787.25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>tolls_amount</th>\n",
       "      <td>7667792.0</td>\n",
       "      <td>0.317</td>\n",
       "      <td>2.024</td>\n",
       "      <td>-70.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>3288.00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>improvement_surcharge</th>\n",
       "      <td>7667792.0</td>\n",
       "      <td>0.299</td>\n",
       "      <td>0.019</td>\n",
       "      <td>-0.3</td>\n",
       "      <td>0.30</td>\n",
       "      <td>0.30</td>\n",
       "      <td>0.30</td>\n",
       "      <td>0.60</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>total_amount</th>\n",
       "      <td>7667792.0</td>\n",
       "      <td>15.682</td>\n",
       "      <td>262.293</td>\n",
       "      <td>-362.8</td>\n",
       "      <td>8.19</td>\n",
       "      <td>11.27</td>\n",
       "      <td>16.56</td>\n",
       "      <td>623261.66</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>congestion_surcharge</th>\n",
       "      <td>2811814.0</td>\n",
       "      <td>0.000</td>\n",
       "      <td>0.009</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>0.00</td>\n",
       "      <td>2.50</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           count     mean      std    min     25%     50%  \\\n",
       "VendorID               7667792.0    1.637    0.540    1.0    1.00    2.00   \n",
       "passenger_count        7667792.0    1.567    1.224    0.0    1.00    1.00   \n",
       "trip_distance          7667792.0    2.801    3.738    0.0    0.90    1.53   \n",
       "RatecodeID             7667792.0    1.058    0.678    1.0    1.00    1.00   \n",
       "PULocationID           7667792.0  165.501   66.392    1.0  130.00  162.00   \n",
       "DOLocationID           7667792.0  163.753   70.364    1.0  113.00  162.00   \n",
       "payment_type           7667792.0    1.292    0.473    1.0    1.00    1.00   \n",
       "fare_amount            7667792.0   12.409  262.072 -362.0    6.00    8.50   \n",
       "extra                  7667792.0    0.328    0.507  -60.0    0.00    0.00   \n",
       "mta_tax                7667792.0    0.497    0.053   -0.5    0.50    0.50   \n",
       "tip_amount             7667792.0    1.827    2.501  -63.5    0.00    1.43   \n",
       "tolls_amount           7667792.0    0.317    2.024  -70.0    0.00    0.00   \n",
       "improvement_surcharge  7667792.0    0.299    0.019   -0.3    0.30    0.30   \n",
       "total_amount           7667792.0   15.682  262.293 -362.8    8.19   11.27   \n",
       "congestion_surcharge   2811814.0    0.000    0.009    0.0    0.00    0.00   \n",
       "\n",
       "                          75%        max  \n",
       "VendorID                 2.00       4.00  \n",
       "passenger_count          2.00       9.00  \n",
       "trip_distance            2.80     831.80  \n",
       "RatecodeID               1.00      99.00  \n",
       "PULocationID           234.00     265.00  \n",
       "DOLocationID           234.00     265.00  \n",
       "payment_type             2.00       4.00  \n",
       "fare_amount             13.50  623259.86  \n",
       "extra                    0.50     535.38  \n",
       "mta_tax                  0.50      60.80  \n",
       "tip_amount               2.33     787.25  \n",
       "tolls_amount             0.00    3288.00  \n",
       "improvement_surcharge    0.30       0.60  \n",
       "total_amount            16.56  623261.66  \n",
       "congestion_surcharge     0.00       2.50  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time \n",
    "np.round(taxi.describe(), 3).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Feature engineering\n",
    "\n",
    "We're going to predict \"high-tip\" rides, meaning rides where the tip percentage is greater than 20%. Tip is defined as the `tip_amount / fare_amount`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "numeric_feat = [\n",
    "    'pickup_weekday', \n",
    "    'pickup_hour', \n",
    "    'pickup_week_hour', \n",
    "    'pickup_minute', \n",
    "    'passenger_count',\n",
    "]\n",
    "categorical_feat = [\n",
    "    'PULocationID', \n",
    "    'DOLocationID',\n",
    "]\n",
    "features = numeric_feat + categorical_feat\n",
    "y_col = 'high_tip'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def prep_df(df: pd.DataFrame) -> pd.DataFrame:\n",
    "    '''\n",
    "    Generate features from a raw taxi dataframe.\n",
    "    '''\n",
    "    df = df[df.fare_amount > 0]  # avoid divide-by-zero\n",
    "    df['tip_fraction'] = df.tip_amount / df.fare_amount\n",
    "    df['high_tip'] = (df['tip_fraction'] > 0.2) # class label\n",
    "    \n",
    "    df['pickup_weekday'] = df.tpep_pickup_datetime.dt.weekday\n",
    "    df['pickup_weekofyear'] = df.tpep_pickup_datetime.dt.weekofyear\n",
    "    df['pickup_hour'] = df.tpep_pickup_datetime.dt.hour\n",
    "    df['pickup_week_hour'] = (df.pickup_weekday * 24) + df.pickup_hour\n",
    "    df['pickup_minute'] = df.tpep_pickup_datetime.dt.minute\n",
    "    df = df[features + [y_col]].astype(float).fillna(-1)\n",
    "    \n",
    "    return df\n",
    "    \n",
    "taxi = prep_df(taxi)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>pickup_weekday</th>\n",
       "      <th>pickup_hour</th>\n",
       "      <th>pickup_week_hour</th>\n",
       "      <th>pickup_minute</th>\n",
       "      <th>passenger_count</th>\n",
       "      <th>PULocationID</th>\n",
       "      <th>DOLocationID</th>\n",
       "      <th>high_tip</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>46.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>151.0</td>\n",
       "      <td>239.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>24.0</td>\n",
       "      <td>59.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>239.0</td>\n",
       "      <td>246.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.0</td>\n",
       "      <td>13.0</td>\n",
       "      <td>109.0</td>\n",
       "      <td>48.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>236.0</td>\n",
       "      <td>236.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>63.0</td>\n",
       "      <td>52.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>193.0</td>\n",
       "      <td>193.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2.0</td>\n",
       "      <td>15.0</td>\n",
       "      <td>63.0</td>\n",
       "      <td>56.0</td>\n",
       "      <td>5.0</td>\n",
       "      <td>193.0</td>\n",
       "      <td>193.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   pickup_weekday  pickup_hour  pickup_week_hour  pickup_minute  \\\n",
       "0             1.0          0.0              24.0           46.0   \n",
       "1             1.0          0.0              24.0           59.0   \n",
       "2             4.0         13.0             109.0           48.0   \n",
       "3             2.0         15.0              63.0           52.0   \n",
       "4             2.0         15.0              63.0           56.0   \n",
       "\n",
       "   passenger_count  PULocationID  DOLocationID  high_tip  \n",
       "0              1.0         151.0         239.0       1.0  \n",
       "1              1.0         239.0         246.0       0.0  \n",
       "2              3.0         236.0         236.0       0.0  \n",
       "3              5.0         193.0         193.0       0.0  \n",
       "4              5.0         193.0         193.0       0.0  "
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "taxi.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Machine learning\n",
    "\n",
    "We'll cover two different machine learning use cases that have differing resource constraints. First, a compute-bound problem: hyperparameter tuning with a linear model on a small dataset. Then, a compute- _and_ memory-bound problem: random forest on a large dataset. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Hyperparameter tuning \n",
    "\n",
    "Grid search with a logistic regression model. We'll sample down the data to ensure this is a compute-bound problem. \n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(382912, 8)"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "taxi_sample = taxi.sample(frac=0.05, replace=False, random_state=seed)\n",
    "taxi_sample.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set up the pipeline and grid search. Setting `n_jobs=-1` tells scikit-learn to use all available cores on this machine to train models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.compose import ColumnTransformer\n",
    "from sklearn.preprocessing import StandardScaler, OneHotEncoder\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "lr = LogisticRegression(\n",
    "    solver='saga',\n",
    "    penalty='elasticnet', \n",
    "    l1_ratio=0.5,\n",
    "    max_iter=100, \n",
    "    random_state=seed,\n",
    ")\n",
    "\n",
    "pipeline = Pipeline(steps=[\n",
    "    ('preprocess', ColumnTransformer(transformers=[\n",
    "        ('num', StandardScaler(), numeric_feat),\n",
    "        ('cat', OneHotEncoder(handle_unknown='ignore', sparse=False), categorical_feat),\n",
    "    ])),\n",
    "    ('clf', lr),\n",
    "])\n",
    "\n",
    "params = {\n",
    "    'clf__l1_ratio': [0.2, 0.3, 0.5, 0.7, 0.9],\n",
    "}\n",
    "\n",
    "grid_search = GridSearchCV(\n",
    "    pipeline, \n",
    "    params, \n",
    "    cv=3, \n",
    "    n_jobs=-1, \n",
    "    verbose=1,\n",
    "    scoring='accuracy',\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3min 44s, sys: 1.11 s, total: 3min 45s\n",
      "Wall time: 3min 46s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "_ = pipeline.fit(taxi_sample[features], taxi_sample[y_col])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 5 candidates, totalling 15 fits\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[Parallel(n_jobs=-1)]: Using backend LokyBackend with 12 concurrent workers.\n",
      "[Parallel(n_jobs=-1)]: Done   8 out of  15 | elapsed:  5.0min remaining:  4.4min\n",
      "[Parallel(n_jobs=-1)]: Done  15 out of  15 | elapsed:  7.7min finished\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 3min 35s, sys: 1.44 s, total: 3min 36s\n",
      "Wall time: 11min 17s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.5612882291887004"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "_ = grid_search.fit(taxi_sample[features], taxi_sample[y_col])\n",
    "grid_search.best_score_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Random forest\n",
    "\n",
    "We're only training one model in this case, so we'll do a train/test split from the full data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2.09 s, sys: 687 ms, total: 2.77 s\n",
      "Wall time: 2.77 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    taxi[features], taxi[y_col], test_size=0.33, random_state=seed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setting `n_jobs=-1` tells scikit-learn to use all available cores on this machine to train the forest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "rfc = RandomForestClassifier(\n",
    "    n_estimators=100, \n",
    "    max_depth=5, \n",
    "    random_state=seed,\n",
    "    n_jobs=-1,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 24min 31s, sys: 18.5 s, total: 24min 50s\n",
      "Wall time: 2min 29s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "_ = rfc.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 22.5 s, sys: 663 ms, total: 23.2 s\n",
      "Wall time: 3.16 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.5501763462272633"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "from sklearn.metrics import roc_auc_score\n",
    "\n",
    "# get test metrics\n",
    "preds = rfc.predict_proba(X_test)[:, 1]\n",
    "roc_auc_score(y_test, preds)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<br>\n",
    "\n",
    "# Moar data!!\n",
    "\n",
    "That wasn't so bad right?? Time for more data!\n",
    "\n",
    "![](https://i.chzbgr.com/full/6993318656/hC83012C2/analyze-all-the-data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['nyc-tlc/trip data/yellow_tripdata_2019-01.csv',\n",
       " 'nyc-tlc/trip data/yellow_tripdata_2019-02.csv',\n",
       " 'nyc-tlc/trip data/yellow_tripdata_2019-03.csv',\n",
       " 'nyc-tlc/trip data/yellow_tripdata_2019-04.csv',\n",
       " 'nyc-tlc/trip data/yellow_tripdata_2019-05.csv',\n",
       " 'nyc-tlc/trip data/yellow_tripdata_2019-06.csv',\n",
       " 'nyc-tlc/trip data/yellow_tripdata_2019-07.csv',\n",
       " 'nyc-tlc/trip data/yellow_tripdata_2019-08.csv',\n",
       " 'nyc-tlc/trip data/yellow_tripdata_2019-09.csv',\n",
       " 'nyc-tlc/trip data/yellow_tripdata_2019-10.csv',\n",
       " 'nyc-tlc/trip data/yellow_tripdata_2019-11.csv',\n",
       " 'nyc-tlc/trip data/yellow_tripdata_2019-12.csv']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "files_2019"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 4min 25s, sys: 36.9 s, total: 5min 2s\n",
      "Wall time: 16min 1s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "dfs = []\n",
    "for f in files_2019:\n",
    "    df = pd.read_csv(\n",
    "        fs.open(f's3://{f}'),\n",
    "        parse_dates=['tpep_pickup_datetime', 'tpep_dropoff_datetime'],\n",
    "    )\n",
    "    dfs.append(df)\n",
    "taxi_2019 = pd.concat(dfs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now we wait...\n",
    "\n",
    "![](https://i.kym-cdn.com/entries/icons/original/000/010/437/Oneeternitylater.jpg)\n",
    "\n",
    "It may eventually complete if your laptop has enough swap space. Worst case, you will run out of memory and your Jupyter kernel will die.\n",
    "\n",
    "Which might leave you feeling a little like...\n",
    "\n",
    "![](https://memegenerator.net/img/instances/61402104.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Do not fear!\n",
    "\n",
    "We're just getting started! Check out [dask.ipynb](dask.ipynb) to see how we can analyze all the files even if they don't fit in RAM."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}