10. Keras Wide and Deep.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# Loading in the Data\n",
    "In this example, we are going to use crossed columns and embedding columns inside of a tensorflow object created with keras.\n",
    "\n",
    "However, we will start the process by loading up a dataset with a mix of categorical data and numeric data. This dataset is quite old and has been used many times in machine learning examples: the census data from 1990's. We will use it to predict if a person will earn over or under 50k per year.\n",
    "\n",
    "- https://archive.ics.uci.edu/ml/datasets/Census-Income+(KDD)\n",
    "- This data goes by various name, including the \"Adult\" dataset and 1996 census data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 32561 entries, 0 to 32560\n",
      "Data columns (total 15 columns):\n",
      " #   Column           Non-Null Count  Dtype \n",
      "---  ------           --------------  ----- \n",
      " 0   age              32561 non-null  int64 \n",
      " 1   workclass        32561 non-null  object\n",
      " 2   fnlwgt           32561 non-null  int64 \n",
      " 3   education        32561 non-null  object\n",
      " 4   edu_num          32561 non-null  int64 \n",
      " 5   marital_status   32561 non-null  object\n",
      " 6   occupation       32561 non-null  object\n",
      " 7   relationship     32561 non-null  object\n",
      " 8   race             32561 non-null  object\n",
      " 9   sex              32561 non-null  object\n",
      " 10  cap_gain         32561 non-null  int64 \n",
      " 11  cap_loss         32561 non-null  int64 \n",
      " 12  work_hrs_weekly  32561 non-null  int64 \n",
      " 13  country          32561 non-null  object\n",
      " 14  income           32561 non-null  object\n",
      "dtypes: int64(6), object(9)\n",
      "memory usage: 3.7+ MB\n",
      "None\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "headers = ['age','workclass','fnlwgt','education','edu_num','marital_status',\n",
    "           'occupation','relationship','race','sex','cap_gain','cap_loss','work_hrs_weekly','country','income']\n",
    "df_train_orig = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data',names=headers)\n",
    "df_test_orig = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.test',names=headers)\n",
    "df_test_orig = df_test_orig.iloc[1:]\n",
    "print(df_train_orig.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "from copy import deepcopy\n",
    "df_train = deepcopy(df_train_orig)\n",
    "df_test = deepcopy(df_test_orig)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The data is organized as follows: \n",
    "\n",
    "|Variable | description|\n",
    "|----|--------|\n",
    "|age: | continuous|\n",
    "|workclass:      |Private, Self-emp-not-inc, Self-emp-inc, Federal-gov, ...|\n",
    "|fnlwgt:         |continuous.|\n",
    "|education:      |Bachelors, Some-college, 11th, HS-grad, Prof-school, ...|\n",
    "|education-num:  |continuous.|\n",
    "|marital-status: |Married-civ-spouse, Divorced, Never-married, Separated, Widowed, ... |\n",
    "|occupation:     |Tech-support, Craft-repair, Other-service, ...|\n",
    "|relationship:   | Wife, Own-child, Husband, Not-in-family, Other-relative, Unmarried.|\n",
    "|race:           |White, Asian-Pac-Islander, Amer-Indian-Eskimo, Other, Black.|\n",
    "|sex:            |Female, Male.|\n",
    "|capital-gain:   |continuous.|\n",
    "|capital-loss:   |continuous.|\n",
    "|hours-per-week: |continuous.|\n",
    "|native-country: |United-States, Cambodia, England, ... |\n",
    "|**income (target):**|<50k, >=50k| "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education</th>\n",
       "      <th>edu_num</th>\n",
       "      <th>marital_status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>race</th>\n",
       "      <th>sex</th>\n",
       "      <th>cap_gain</th>\n",
       "      <th>cap_loss</th>\n",
       "      <th>work_hrs_weekly</th>\n",
       "      <th>country</th>\n",
       "      <th>income</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>25</td>\n",
       "      <td>Private</td>\n",
       "      <td>226802.0</td>\n",
       "      <td>11th</td>\n",
       "      <td>7.0</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Machine-op-inspct</td>\n",
       "      <td>Own-child</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>38</td>\n",
       "      <td>Private</td>\n",
       "      <td>89814.0</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9.0</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Farming-fishing</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>50.0</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>28</td>\n",
       "      <td>Local-gov</td>\n",
       "      <td>336951.0</td>\n",
       "      <td>Assoc-acdm</td>\n",
       "      <td>12.0</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Protective-serv</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&gt;50K.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>44</td>\n",
       "      <td>Private</td>\n",
       "      <td>160323.0</td>\n",
       "      <td>Some-college</td>\n",
       "      <td>10.0</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Machine-op-inspct</td>\n",
       "      <td>Husband</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>7688.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>40.0</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&gt;50K.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>34</td>\n",
       "      <td>Private</td>\n",
       "      <td>198693.0</td>\n",
       "      <td>10th</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Other-service</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>30.0</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  age   workclass    fnlwgt      education  edu_num       marital_status   \n",
       "1  25     Private  226802.0           11th      7.0        Never-married  \\\n",
       "2  38     Private   89814.0        HS-grad      9.0   Married-civ-spouse   \n",
       "3  28   Local-gov  336951.0     Assoc-acdm     12.0   Married-civ-spouse   \n",
       "4  44     Private  160323.0   Some-college     10.0   Married-civ-spouse   \n",
       "6  34     Private  198693.0           10th      6.0        Never-married   \n",
       "\n",
       "           occupation    relationship    race    sex  cap_gain  cap_loss   \n",
       "1   Machine-op-inspct       Own-child   Black   Male       0.0       0.0  \\\n",
       "2     Farming-fishing         Husband   White   Male       0.0       0.0   \n",
       "3     Protective-serv         Husband   White   Male       0.0       0.0   \n",
       "4   Machine-op-inspct         Husband   Black   Male    7688.0       0.0   \n",
       "6       Other-service   Not-in-family   White   Male       0.0       0.0   \n",
       "\n",
       "   work_hrs_weekly         country   income  \n",
       "1             40.0   United-States   <=50K.  \n",
       "2             50.0   United-States   <=50K.  \n",
       "3             40.0   United-States    >50K.  \n",
       "4             40.0   United-States    >50K.  \n",
       "6             30.0   United-States   <=50K.  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "# let's just get rid of rows with any missing data\n",
    "# and then reset the indices of the dataframe so it corresponds to row number\n",
    "df_train.replace(to_replace=' ?',value=np.nan, inplace=True)\n",
    "df_train.dropna(inplace=True)\n",
    "df_train.reset_index()\n",
    "\n",
    "df_test.replace(to_replace=' ?',value=np.nan, inplace=True)\n",
    "df_test.dropna(inplace=True)\n",
    "df_test.reset_index()\n",
    "\n",
    "df_test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Processing\n",
    "For preprocessing, we are going to fix a few issues in the dataset. \n",
    "\n",
    "- This first includes the use of \"50K.\" instead of \"50K\" in the test set. \n",
    "- Next, we will encode the categorical features as integers (eventually we will deal with these integers like a one-hot encoding)\n",
    "- Finally, we will make certain all the continuous data is scaled properly with zero mean and unit standard deviaiton"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "income\n",
      "<=50K    11360\n",
      ">50K      3700\n",
      "Name: count, dtype: int64\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education</th>\n",
       "      <th>edu_num</th>\n",
       "      <th>marital_status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>race</th>\n",
       "      <th>sex</th>\n",
       "      <th>...</th>\n",
       "      <th>country</th>\n",
       "      <th>income</th>\n",
       "      <th>workclass_int</th>\n",
       "      <th>education_int</th>\n",
       "      <th>marital_status_int</th>\n",
       "      <th>occupation_int</th>\n",
       "      <th>relationship_int</th>\n",
       "      <th>race_int</th>\n",
       "      <th>sex_int</th>\n",
       "      <th>country_int</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.023104</td>\n",
       "      <td>Private</td>\n",
       "      <td>226802.0</td>\n",
       "      <td>11th</td>\n",
       "      <td>7.0</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Machine-op-inspct</td>\n",
       "      <td>Own-child</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>...</td>\n",
       "      <td>United-States</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.033340</td>\n",
       "      <td>Private</td>\n",
       "      <td>89814.0</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9.0</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Farming-fishing</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>...</td>\n",
       "      <td>United-States</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>11</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.794697</td>\n",
       "      <td>Local-gov</td>\n",
       "      <td>336951.0</td>\n",
       "      <td>Assoc-acdm</td>\n",
       "      <td>12.0</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Protective-serv</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>...</td>\n",
       "      <td>United-States</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>2</td>\n",
       "      <td>10</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.423474</td>\n",
       "      <td>Private</td>\n",
       "      <td>160323.0</td>\n",
       "      <td>Some-college</td>\n",
       "      <td>10.0</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Machine-op-inspct</td>\n",
       "      <td>Husband</td>\n",
       "      <td>Black</td>\n",
       "      <td>Male</td>\n",
       "      <td>...</td>\n",
       "      <td>United-States</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>15</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>-0.337883</td>\n",
       "      <td>Private</td>\n",
       "      <td>198693.0</td>\n",
       "      <td>10th</td>\n",
       "      <td>6.0</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Other-service</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>...</td>\n",
       "      <td>United-States</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>7</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>38</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 23 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        age  workclass    fnlwgt     education  edu_num      marital_status   \n",
       "1 -1.023104    Private  226802.0          11th      7.0       Never-married  \\\n",
       "2 -0.033340    Private   89814.0       HS-grad      9.0  Married-civ-spouse   \n",
       "3 -0.794697  Local-gov  336951.0    Assoc-acdm     12.0  Married-civ-spouse   \n",
       "4  0.423474    Private  160323.0  Some-college     10.0  Married-civ-spouse   \n",
       "6 -0.337883    Private  198693.0          10th      6.0       Never-married   \n",
       "\n",
       "          occupation   relationship   race   sex  ...        country  income   \n",
       "1  Machine-op-inspct      Own-child  Black  Male  ...  United-States       0  \\\n",
       "2    Farming-fishing        Husband  White  Male  ...  United-States       0   \n",
       "3    Protective-serv        Husband  White  Male  ...  United-States       1   \n",
       "4  Machine-op-inspct        Husband  Black  Male  ...  United-States       1   \n",
       "6      Other-service  Not-in-family  White  Male  ...  United-States       0   \n",
       "\n",
       "   workclass_int education_int  marital_status_int  occupation_int   \n",
       "1              2             1                   4               6  \\\n",
       "2              2            11                   2               4   \n",
       "3              1             7                   2              10   \n",
       "4              2            15                   2               6   \n",
       "6              2             0                   4               7   \n",
       "\n",
       "   relationship_int  race_int  sex_int  country_int  \n",
       "1                 3         2        1           38  \n",
       "2                 0         4        1           38  \n",
       "3                 0         4        1           38  \n",
       "4                 0         2        1           38  \n",
       "6                 1         4        1           38  \n",
       "\n",
       "[5 rows x 23 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.preprocessing import LabelEncoder\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# replace weird strings inconsistency\n",
    "if df_test.income.dtype=='object':\n",
    "    df_test.income.replace(to_replace=[' <=50K.',' >50K.'],\n",
    "                           value=['<=50K','>50K'],\n",
    "                           inplace=True)\n",
    "    print(df_test.income.value_counts())\n",
    "\n",
    "# ========================================================\n",
    "# define objects that can encode each variable as integer    \n",
    "encoders = dict() # save each encoder in dictionary\n",
    "categorical_headers = ['workclass','education','marital_status',\n",
    "                       'occupation','relationship','race','sex','country']\n",
    "\n",
    "# train all encoders (special case the target 'income')\n",
    "for col in categorical_headers+['income']:\n",
    "    df_train[col] = df_train[col].str.strip()\n",
    "    df_test[col] = df_test[col].str.strip()\n",
    "    \n",
    "    if col==\"income\":\n",
    "        # special case the target, just replace the column\n",
    "        tmp = LabelEncoder()\n",
    "        df_train[col] = tmp.fit_transform(df_train[col])\n",
    "        df_test[col] = tmp.transform(df_test[col])\n",
    "    else:\n",
    "        # integer encode strings that are features\n",
    "        encoders[col] = LabelEncoder() # save the encoder\n",
    "        df_train[col+'_int'] = encoders[col].fit_transform(df_train[col])\n",
    "        df_test[col+'_int'] = encoders[col].transform(df_test[col])\n",
    "\n",
    "# ========================================================\n",
    "# scale the numeric, continuous variables\n",
    "numeric_headers = [\"age\", \"cap_gain\", \"cap_loss\",\"work_hrs_weekly\"]\n",
    "\n",
    "ss = StandardScaler()\n",
    "df_train[numeric_headers] = ss.fit_transform(df_train[numeric_headers].values)\n",
    "df_test[numeric_headers] = ss.transform(df_test[numeric_headers].values)\n",
    "\n",
    "# Code above does the same as the following, for each column:\n",
    "# for col in numeric_headers:\n",
    "#     df_train[col] = df_train[col].astype(np.float)\n",
    "#     df_test[col] = df_test[col].astype(np.float)\n",
    "    \n",
    "#     ss = StandardScaler()\n",
    "#     df_train[col] = ss.fit_transform(df_train[col].values.reshape(-1, 1))\n",
    "#     df_test[col] = ss.transform(df_test[col].values.reshape(-1, 1))\n",
    "    \n",
    "df_test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "We will use the following 12 features:\n",
      "[   'workclass_int',\n",
      "    'education_int',\n",
      "    'marital_status_int',\n",
      "    'occupation_int',\n",
      "    'relationship_int',\n",
      "    'race_int',\n",
      "    'sex_int',\n",
      "    'country_int',\n",
      "    'age',\n",
      "    'cap_gain',\n",
      "    'cap_loss',\n",
      "    'work_hrs_weekly']\n"
     ]
    }
   ],
   "source": [
    "# let's start as simply as possible, without any feature preprocessing\n",
    "categorical_headers_ints = [x+'_int' for x in categorical_headers]\n",
    "\n",
    "# we will forego one-hot encoding right now and instead just use all inputs as-is\n",
    "#   this is just to get an example running in Keras (its not a good idea)\n",
    "feature_columns = categorical_headers_ints+numeric_headers\n",
    "\n",
    "import pprint\n",
    "pp = pprint.PrettyPrinter(indent=4)\n",
    "\n",
    "print(f\"We will use the following {len(feature_columns)} features:\")\n",
    "pp.pprint(feature_columns)\n",
    "\n",
    "# combine the features into a single large matrix\n",
    "X_train = df_train[feature_columns].to_numpy()\n",
    "X_test = df_test[feature_columns].to_numpy()\n",
    "\n",
    "y_train = df_train['income'].values.astype(np.int32)\n",
    "y_test = df_test['income'].values.astype(np.int32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Numeric Headers:\n",
      "['age', 'cap_gain', 'cap_loss', 'work_hrs_weekly']\n",
      "\n",
      "Categorical String Headers:\n",
      "[   'workclass',\n",
      "    'education',\n",
      "    'marital_status',\n",
      "    'occupation',\n",
      "    'relationship',\n",
      "    'race',\n",
      "    'sex',\n",
      "    'country']\n",
      "\n",
      "Categorical Headers, Encoded as Integer:\n",
      "[   'workclass_int',\n",
      "    'education_int',\n",
      "    'marital_status_int',\n",
      "    'occupation_int',\n",
      "    'relationship_int',\n",
      "    'race_int',\n",
      "    'sex_int',\n",
      "    'country_int']\n"
     ]
    }
   ],
   "source": [
    "# we have the following lists now of data that we can use with our dataframes:\n",
    "print(\"Numeric Headers:\")\n",
    "pp.pprint(numeric_headers) # normalized numeric data\n",
    "print(\"\\nCategorical String Headers:\")\n",
    "pp.pprint(categorical_headers) # string data\n",
    "print(\"\\nCategorical Headers, Encoded as Integer:\")\n",
    "pp.pprint(categorical_headers_ints) # string data encoded as an integer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![keras](https://blog.keras.io/img/keras-tensorflow-logo.jpg)\n",
    "\n",
    "\n",
    "# An example similar to Sklearn, Keras\n",
    "- We will start with creating a model that is similar to what we have already written. \n",
    "- Let's not worry about pre-processing the data right now. Let's just get familiar with Keras."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.12.0\n",
      "2.12.0\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "os.environ['AUTOGRAPH_VERBOSITY'] = '0'\n",
    "\n",
    "from sklearn import metrics as mt\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "\n",
    "\n",
    "print(tf.__version__)\n",
    "print(keras.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.layers import Dense, Activation, Input\n",
    "from tensorflow.keras.models import Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Metal device set to: Apple M2\n"
     ]
    }
   ],
   "source": [
    "# Now let's define the architecture for a multi-layer network\n",
    "\n",
    "# First, lets setup the input size\n",
    "num_features = X_train.shape[1]\n",
    "input_tensor = Input(shape=(num_features,))\n",
    "\n",
    "# a layer instance is callable on a tensor, and returns a tensor\n",
    "# Dense means a fully connected layer, with 10 hidden neurons and a bias term\n",
    "x = Dense(units=10, activation='relu')(input_tensor)\n",
    "x = Dense(units=5, activation='tanh')(x)\n",
    "predictions = Dense(1, activation='sigmoid')(x)\n",
    "\n",
    "# This creates a model that includes\n",
    "# the Input layer and three Dense layers\n",
    "model = Model(inputs=input_tensor, outputs=predictions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "___\n",
    "## Adding more customization\n",
    "For this particular optimization, we could go beyond SGD and use a different optimizer. There are many excellent explanations of different optimizers, for instance:\n",
    "- http://sebastianruder.com/optimizing-gradient-descent/\n",
    "\n",
    "Now that we have defined the model, its as simple as using it in a very familiar syntax:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model\"\n",
      "_________________________________________________________________\n",
      " Layer (type)                Output Shape              Param #   \n",
      "=================================================================\n",
      " input_1 (InputLayer)        [(None, 12)]              0         \n",
      "                                                                 \n",
      " dense (Dense)               (None, 10)                130       \n",
      "                                                                 \n",
      " dense_1 (Dense)             (None, 5)                 55        \n",
      "                                                                 \n",
      " dense_2 (Dense)             (None, 1)                 6         \n",
      "                                                                 \n",
      "=================================================================\n",
      "Total params: 191\n",
      "Trainable params: 191\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.compile(optimizer='sgd',\n",
    "              loss='mean_squared_error',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You must install pydot (`pip install pydot`) and install graphviz (see instructions at https://graphviz.gitlab.io/download/) for plot_model to work.\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.utils import plot_model\n",
    "\n",
    "# you will need to install pydot properly on your machine to get this running\n",
    "plot_model(\n",
    "    model, to_file='model.png', show_shapes=True, show_layer_names=True,\n",
    "    rankdir='LR', expand_nested=False, dpi=96\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/10\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "2023-08-11 15:19:06.620348: W tensorflow/tsl/platform/profile_utils/cpu_utils.cc:128] Failed to get CPU frequency: 0 Hz\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "604/604 [==============================] - 2s 3ms/step - loss: 0.1826 - accuracy: 0.7425\n",
      "Epoch 2/10\n",
      "604/604 [==============================] - 2s 3ms/step - loss: 0.1708 - accuracy: 0.7536\n",
      "Epoch 3/10\n",
      "604/604 [==============================] - 2s 3ms/step - loss: 0.1633 - accuracy: 0.7610\n",
      "Epoch 4/10\n",
      "604/604 [==============================] - 2s 3ms/step - loss: 0.1607 - accuracy: 0.7619\n",
      "Epoch 5/10\n",
      "604/604 [==============================] - 2s 3ms/step - loss: 0.1585 - accuracy: 0.7638\n",
      "Epoch 6/10\n",
      "604/604 [==============================] - 2s 3ms/step - loss: 0.1568 - accuracy: 0.7666\n",
      "Epoch 7/10\n",
      "604/604 [==============================] - 2s 3ms/step - loss: 0.1548 - accuracy: 0.7708\n",
      "Epoch 8/10\n",
      "604/604 [==============================] - 2s 3ms/step - loss: 0.1534 - accuracy: 0.7748\n",
      "Epoch 9/10\n",
      "604/604 [==============================] - 2s 3ms/step - loss: 0.1525 - accuracy: 0.7757\n",
      "Epoch 10/10\n",
      "604/604 [==============================] - 2s 4ms/step - loss: 0.1515 - accuracy: 0.7785\n",
      "471/471 [==============================] - 1s 1ms/step\n",
      "[[8280 3080]\n",
      " [1036 2664]]\n",
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.89      0.73      0.80     11360\n",
      "           1       0.46      0.72      0.56      3700\n",
      "\n",
      "    accuracy                           0.73     15060\n",
      "   macro avg       0.68      0.72      0.68     15060\n",
      "weighted avg       0.78      0.73      0.74     15060\n",
      "\n",
      "CPU times: user 16.3 s, sys: 7.94 s, total: 24.2 s\n",
      "Wall time: 21.6 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "model.fit(X_train, y_train, epochs=10, batch_size=50, verbose=1)\n",
    "\n",
    "# now lets see how well the model performed\n",
    "from sklearn import metrics as mt\n",
    "yhat_proba = model.predict(X_test) # sigmoid output probabilities\n",
    "yhat = np.round(yhat_proba) # round to get binary class\n",
    "\n",
    "print(mt.confusion_matrix(y_test,yhat))\n",
    "print(mt.classification_report(y_test,yhat))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ouch! It didn't work very well at all! Sometimes is will not even train better than chance. **But we are not handling categorical data properly here...** Let's fix that.\n",
    "\n",
    "**[back to slides]**\n",
    "\n",
    "## Embedding Categorical Data\n",
    "In order to add one-hot encoding, we need to separate the categorical features that are currently saved as integers and place them into Embedding layers. An embedding layer deals with integers as if they were one-hot encoded. To start, let's only create a model from these categorical variables. \n",
    "\n",
    "- The first decision to make is about the size of the dense feature embeddings. This is essentially a dimensionality reduction step. \n",
    " - This can be difficult to set, but one common setting is $\\log_2(N)$ or $\\sqrt{N}$ where $N$ is the total number of uniques values.\n",
    "- When using an Embedding, we can leave the variables represented as integers. That is, keras will use the integer representation to figure out how to one-hot-encode the variable. This is great because it means we don't need to change the variable in memory.\n",
    "- We will feed all the categorical variables to the model and, internally, the model will separate out the columns and apply a separate embedding. The function we will use to get each column is the `gather` function that is part of tensorflow."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"model_1\"\n",
      "__________________________________________________________________________________________________\n",
      " Layer (type)                   Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      " categorical (InputLayer)       [(None, 8)]          0           []                               \n",
      "                                                                                                  \n",
      " tf.compat.v1.gather (TFOpLambd  (None,)             0           ['categorical[0][0]']            \n",
      " a)                                                                                               \n",
      "                                                                                                  \n",
      " tf.compat.v1.gather_1 (TFOpLam  (None,)             0           ['categorical[0][0]']            \n",
      " bda)                                                                                             \n",
      "                                                                                                  \n",
      " tf.compat.v1.gather_2 (TFOpLam  (None,)             0           ['categorical[0][0]']            \n",
      " bda)                                                                                             \n",
      "                                                                                                  \n",
      " tf.compat.v1.gather_3 (TFOpLam  (None,)             0           ['categorical[0][0]']            \n",
      " bda)                                                                                             \n",
      "                                                                                                  \n",
      " tf.compat.v1.gather_4 (TFOpLam  (None,)             0           ['categorical[0][0]']            \n",
      " bda)                                                                                             \n",
      "                                                                                                  \n",
      " tf.compat.v1.gather_5 (TFOpLam  (None,)             0           ['categorical[0][0]']            \n",
      " bda)                                                                                             \n",
      "                                                                                                  \n",
      " tf.compat.v1.gather_6 (TFOpLam  (None,)             0           ['categorical[0][0]']            \n",
      " bda)                                                                                             \n",
      "                                                                                                  \n",
      " tf.compat.v1.gather_7 (TFOpLam  (None,)             0           ['categorical[0][0]']            \n",
      " bda)                                                                                             \n",
      "                                                                                                  \n",
      " workclass_int_embed (Embedding  (None, 2)           14          ['tf.compat.v1.gather[0][0]']    \n",
      " )                                                                                                \n",
      "                                                                                                  \n",
      " education_int_embed (Embedding  (None, 4)           64          ['tf.compat.v1.gather_1[0][0]']  \n",
      " )                                                                                                \n",
      "                                                                                                  \n",
      " marital_status_int_embed (Embe  (None, 2)           14          ['tf.compat.v1.gather_2[0][0]']  \n",
      " dding)                                                                                           \n",
      "                                                                                                  \n",
      " occupation_int_embed (Embeddin  (None, 3)           42          ['tf.compat.v1.gather_3[0][0]']  \n",
      " g)                                                                                               \n",
      "                                                                                                  \n",
      " relationship_int_embed (Embedd  (None, 2)           12          ['tf.compat.v1.gather_4[0][0]']  \n",
      " ing)                                                                                             \n",
      "                                                                                                  \n",
      " race_int_embed (Embedding)     (None, 2)            10          ['tf.compat.v1.gather_5[0][0]']  \n",
      "                                                                                                  \n",
      " sex_int_embed (Embedding)      (None, 1)            2           ['tf.compat.v1.gather_6[0][0]']  \n",
      "                                                                                                  \n",
      " country_int_embed (Embedding)  (None, 6)            246         ['tf.compat.v1.gather_7[0][0]']  \n",
      "                                                                                                  \n",
      " concat_1 (Concatenate)         (None, 22)           0           ['workclass_int_embed[0][0]',    \n",
      "                                                                  'education_int_embed[0][0]',    \n",
      "                                                                  'marital_status_int_embed[0][0]'\n",
      "                                                                 , 'occupation_int_embed[0][0]',  \n",
      "                                                                  'relationship_int_embed[0][0]', \n",
      "                                                                  'race_int_embed[0][0]',         \n",
      "                                                                  'sex_int_embed[0][0]',          \n",
      "                                                                  'country_int_embed[0][0]']      \n",
      "                                                                                                  \n",
      " combined (Dense)               (None, 1)            23          ['concat_1[0][0]']               \n",
      "                                                                                                  \n",
      "==================================================================================================\n",
      "Total params: 427\n",
      "Trainable params: 427\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.layers import Embedding\n",
    "from tensorflow.keras.layers import concatenate\n",
    "\n",
    "# start by getting only the categorical variables\n",
    "# these matrices are all integers\n",
    "X_train = df_train[categorical_headers_ints].to_numpy() \n",
    "X_test = df_test[categorical_headers_ints].to_numpy() \n",
    "\n",
    "embed_branches = []\n",
    "all_branch_outputs = [] # this is where we will keep track of output of each branch\n",
    "\n",
    "# feed in the entire matrix of categircal variables\n",
    "input_branch = Input(shape=(X_train.shape[1],), \n",
    "                     dtype='int64', \n",
    "                     name='categorical')\n",
    "\n",
    "# for each categorical variable\n",
    "for idx,col in enumerate(categorical_headers_ints):\n",
    "    \n",
    "    # what the maximum integer value for this variable?\n",
    "    # which is the same as the number of categories\n",
    "    # these lines of code do the same thing\n",
    "    N = max(df_train[col].max(),df_test[col].max())+1\n",
    "    N = len(encoders[categorical_headers[idx]].classes_)\n",
    "    \n",
    "    # this line of code does this: input_branch[:,idx]\n",
    "    x = tf.gather(input_branch, idx, axis=1)\n",
    "    \n",
    "    # now use an embedding to deal with integers as if they were one hot encoded\n",
    "    x = Embedding(input_dim=N, \n",
    "                  output_dim=int(np.sqrt(N)), \n",
    "                  input_length=1, name=col+'_embed')(x)\n",
    "    \n",
    "    # save these outputs in list to concatenate later\n",
    "    all_branch_outputs.append(x)\n",
    "    \n",
    "# now concatenate the outputs and add a fully connected layer\n",
    "final_branch = concatenate(all_branch_outputs, name='concat_1')\n",
    "final_branch = Dense(units=1,\n",
    "                     activation='sigmoid', \n",
    "                     name='combined')(final_branch)\n",
    "\n",
    "model = Model(inputs=input_branch, outputs=final_branch)\n",
    "\n",
    "model.compile(optimizer='sgd',\n",
    "              loss='mean_squared_error',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "You must install pydot (`pip install pydot`) and install graphviz (see instructions at https://graphviz.gitlab.io/download/) for plot_model to work.\n"
     ]
    }
   ],
   "source": [
    "plot_model(\n",
    "    model, to_file='model.png', show_shapes=True, show_layer_names=True,\n",
    "    rankdir='LR', expand_nested=False, dpi=96\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "model.fit(X_train, y_train, epochs=10, batch_size=50, verbose=1)\n",
    "\n",
    "from sklearn import metrics as mt\n",
    "yhat_proba = model.predict(X_test)\n",
    "yhat = np.round(yhat_proba)\n",
    "print(mt.confusion_matrix(y_test,yhat))\n",
    "print(mt.classification_report(y_test,yhat))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Much better!! Looks like the use of Embedding layers helped. Lets add some more of the features, using both the categorical and the integer features.\n",
    "\n",
    "## Multi-Modal Networks: Combine Branches, Sparse and Dense\n",
    "The keras functional API allows us to setup different branches of inputs. When we call the `fit` function, we can tell keras the input to each branch using a simple list (you can also use a dictionary to be more explicit).\n",
    "\n",
    "- Lets create two matrices: \n",
    " - one with the categorical data, `X_train_cat`\n",
    " - and another with the numeric data, `X_train_num`\n",
    " \n",
    "Now, we can feed both of these into the model as a list and concatenate their results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# save categorical features\n",
    "X_train_cat = df_train[categorical_headers_ints].to_numpy() \n",
    "X_test_cat = df_test[categorical_headers_ints].to_numpy() \n",
    "\n",
    "# and save off the numeric features\n",
    "X_train_num =  df_train[numeric_headers].to_numpy()\n",
    "X_test_num = df_test[numeric_headers].to_numpy()\n",
    "\n",
    "all_branch_outputs = [] # this is where we will keep track of output of each branch\n",
    "\n",
    "# -----Like previous example, use embeddings from the integer data -----\n",
    "input_cat = Input(shape=(X_train_cat.shape[1],), dtype='int64', name='categorical')\n",
    "for idx,col in enumerate(categorical_headers_ints):\n",
    "    \n",
    "    # track what the maximum integer value will be for this variable\n",
    "    # which is the same as the number of categories\n",
    "    N = max(df_train[col].max(),df_test[col].max())+1\n",
    "    N = len(encoders[categorical_headers[idx]].classes_)\n",
    "    N_reduced = int(np.sqrt(N))\n",
    "    \n",
    "    # this line of code does this: input_branch[:,idx]\n",
    "    x = tf.gather(input_cat, idx, axis=1)\n",
    "    \n",
    "    # now use an embedding to deal with integers as if they were one hot encoded\n",
    "    x = Embedding(input_dim=N, \n",
    "                  output_dim=N_reduced, \n",
    "                  input_length=1, name=col+'_embed')(x)\n",
    "    \n",
    "    # save these outputs to concatenate later\n",
    "    all_branch_outputs.append(x)\n",
    "    \n",
    "# -----HERE IS THE ADDING OF AN INPUT USING NUMERIC DATA-----\n",
    "# create dense input branch for numeric\n",
    "inputs_num = Input(shape=(X_train_num.shape[1],), name='numeric')\n",
    "x_dense = Dense(units=9, activation='relu', name='num_1')(inputs_num)\n",
    "    \n",
    "all_branch_outputs.append(x_dense)\n",
    "# -------------------------------------------------------------------\n",
    "\n",
    "# now concatenate the outputs and add a fully connected layer\n",
    "final_branch = concatenate(all_branch_outputs, name='concat_1')\n",
    "final_branch = Dense(units=10, activation='relu', name='combined_1')(final_branch)\n",
    "final_branch = Dense(units=1, activation='sigmoid', name='combined_2')(final_branch)\n",
    "\n",
    "model = Model(inputs=[input_cat,inputs_num], outputs=final_branch)\n",
    "\n",
    "model.compile(optimizer='sgd',\n",
    "              loss='mean_squared_error',\n",
    "              metrics=['accuracy'])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_model(\n",
    "    model, to_file='model.png', show_shapes=True, show_layer_names=True,\n",
    "    rankdir='LR', expand_nested=False, dpi=96\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "model.compile(optimizer='sgd',\n",
    "              loss='mean_squared_error',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit([ X_train_cat, X_train_num ], # inputs for each branch are a list\n",
    "          y_train, \n",
    "          epochs=10, \n",
    "          batch_size=50, \n",
    "          verbose=1)\n",
    "\n",
    "yhat = model.predict([X_test_cat,\n",
    "                      X_test_num]) # each branch has an input\n",
    "\n",
    "yhat = np.round(yhat)\n",
    "print(mt.confusion_matrix(y_test,yhat))\n",
    "print(mt.classification_report(y_test,yhat))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So the confusion matrix is doing better! But we still are just using a simple neural network. We really want to take advantage of the embeddings and crossed columns that are possible with tensorflow. \n",
    "\n",
    "\n",
    "**[Back to Slides]**\n",
    "___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Crossed Columns and Deep Embeddings\n",
    "Lets create the network below in steps. This network consists of a number of different paths for the input data to take.\n",
    "![asdfasfd](https://www.tensorflow.org/images/wide_n_deep.svg)\n",
    "\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "___\n",
    "\n",
    "\n",
    "## Making Crossed Columns\n",
    "For this example, we are going to make embeddings for crossed columns. We will perform this step using `sklearn` \n",
    "\n",
    "Let's start simple with \n",
    "- branches of crossed categorical features as input \n",
    "- one branch for each crossed column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sandbox for looking at different categorical variables\n",
    "for col in categorical_headers:\n",
    "    vals = df_train[col].unique()\n",
    "    print(col,'has', len(vals), 'unique values:')\n",
    "    print(vals)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# a quick example of crossing some columns\n",
    "\n",
    "cols_list = ['marital_status','workclass','relationship']\n",
    "\n",
    "# 1. create crossed labels by string join operation\n",
    "X_crossed_train = df_train[cols_list].apply(lambda x: '_'.join(x), axis=1)\n",
    "X_crossed_test = df_test[cols_list].apply(lambda x: '_'.join(x), axis=1)\n",
    "\n",
    "# combine together for training\n",
    "all_vals = np.hstack((X_crossed_train.to_numpy(),  X_crossed_test.to_numpy()))\n",
    "print(np.unique(all_vals))\n",
    "    \n",
    "# 2. encode as integers, stacking all possibilities\n",
    "enc = LabelEncoder()\n",
    "enc.fit(all_vals)\n",
    "\n",
    "encoded_vals_train = enc.transform(X_crossed_train)\n",
    "encoded_vals_test  = enc.transform(X_crossed_test)\n",
    "\n",
    "print(np.min(encoded_vals_train), np.max(encoded_vals_train))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# now let's create some different crossed values\n",
    "# possible crossing options:\n",
    "#   'workclass','education','marital_status',\n",
    "#   'occupation','relationship','race',\n",
    "#   'sex','country'\n",
    "\n",
    "# choose these as a class, what makes sense??\n",
    "cross_columns = [#['race','sex','education','occupation'],\n",
    "                 ['race','sex', 'marital_status','occupation'],\n",
    "                 ['workclass','occupation'],\n",
    "                 ['workclass','occupation','education'],\n",
    "                 #['occupation','race','education'],\n",
    "                 ['education','country']\n",
    "                ]\n",
    "\n",
    "# cross each set of columns in the list above\n",
    "cross_col_df_names = []\n",
    "for cols_list in cross_columns:\n",
    "    # encode as ints for the embedding\n",
    "    enc = LabelEncoder()\n",
    "    \n",
    "    # 1. create crossed labels by join operation\n",
    "    X_crossed_train = df_train[cols_list].apply(lambda x: '_'.join(x), axis=1)\n",
    "    X_crossed_test = df_test[cols_list].apply(lambda x: '_'.join(x), axis=1)\n",
    "    \n",
    "    # get a nice name for this new crossed column\n",
    "    cross_col_name = '_'.join(cols_list)\n",
    "    \n",
    "    # 2. encode as integers, stacking all possibilities\n",
    "    enc.fit(np.hstack((X_crossed_train.to_numpy(),  X_crossed_test.to_numpy())))\n",
    "    \n",
    "    # 3. Save into dataframe with new name\n",
    "    df_train[cross_col_name] = enc.transform(X_crossed_train)\n",
    "    df_test[cross_col_name] = enc.transform(X_crossed_test)\n",
    "    \n",
    "    # Save the encoder used here for later:\n",
    "    encoders[cross_col_name] = enc\n",
    "    \n",
    "    # keep track of the new names of the crossed columns\n",
    "    cross_col_df_names.append(cross_col_name) \n",
    "    \n",
    "cross_col_df_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Train a model only using crossed values\n",
    "# get crossed columns\n",
    "X_train_crossed = df_train[cross_col_df_names].to_numpy()\n",
    "X_test_crossed = df_test[cross_col_df_names].to_numpy()\n",
    "\n",
    "crossed_outputs = [] # this is where we will keep track of output of each branch\n",
    "\n",
    "input_crossed = Input(shape=(X_train_crossed.shape[1],), dtype='int64', name='categorical')\n",
    "for idx,col in enumerate(cross_col_df_names):\n",
    "    \n",
    "    # track what the maximum integer value will be for this variable\n",
    "    # which is the same as the number of categories\n",
    "    N = max(df_train[col].max(),df_test[col].max())+1\n",
    "    N = len(encoders[col].classes_)\n",
    "    N_reduced = int(np.sqrt(N))\n",
    "    \n",
    "    # this line of code does this: input_branch[:,idx]\n",
    "    x = tf.gather(input_crossed, idx, axis=1)\n",
    "    \n",
    "    # now use an embedding to deal with integers as if they were one hot encoded\n",
    "    x = Embedding(input_dim=N, \n",
    "                  output_dim=N_reduced, \n",
    "                  input_length=1, name=col+'_embed')(x)\n",
    "    \n",
    "    # save these outputs to concatenate later\n",
    "    crossed_outputs.append(x)\n",
    "    \n",
    "\n",
    "# now concatenate the outputs and add a fully connected layer\n",
    "wide_branch = concatenate(crossed_outputs, name='concat_1')\n",
    "wide_branch = Dense(units=1,activation='sigmoid', name='combined')(wide_branch)\n",
    "\n",
    "model = Model(inputs=input_crossed, outputs=wide_branch)\n",
    "\n",
    "model.compile(optimizer='sgd',\n",
    "              loss='mean_squared_error',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "model.fit(X_train_crossed,\n",
    "        y_train, epochs=10, batch_size=32, verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# bonus: we can keep training, picking up where we left of (no reinitialization)\n",
    "model.fit(X_train_crossed,\n",
    "        y_train, epochs=10, batch_size=32, verbose=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "yhat = np.round(model.predict(X_test_crossed))\n",
    "print(mt.confusion_matrix(y_test,yhat))\n",
    "print(mt.classification_report(y_test,yhat))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you will need to install pydot properly on your machine to get this running\n",
    "plot_model(\n",
    "    model, to_file='model.png', show_shapes=True, show_layer_names=True,\n",
    "    rankdir='LR', expand_nested=False, dpi=96\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wow! That is just using crossed columns and a one layer Linear Classifer! So memorization works fairly well here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "___\n",
    "\n",
    "\n",
    "## Combining Crossed Linear Classifier and Deep Embeddings\n",
    "Now its just a matter of setting the wide and deep columns for tensorflow. After which, we can use the combined classifier!\n",
    "\n",
    "Wide and deep models can have really interesting and useful properties so they are great to keep in mind when selecting an architecture. Some of the hyperparameters that are specific to this are:\n",
    "- which features to cross together, typically you only want to cross columns you think are important to be connected--they somehow might create new knowledge by combining.\n",
    "- the size of the dense feature embeddings. This can be difficult to set, but one common setting is $log_2(N)$ or $\\sqrt{N}$ where $N$ is the total number of uniques values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get crossed columns\n",
    "X_train_crossed = df_train[cross_col_df_names].to_numpy()\n",
    "X_test_crossed = df_test[cross_col_df_names].to_numpy()\n",
    "\n",
    "# save categorical features\n",
    "X_train_cat = df_train[categorical_headers_ints].to_numpy() \n",
    "X_test_cat = df_test[categorical_headers_ints].to_numpy() \n",
    "\n",
    "# and save off the numeric features\n",
    "X_train_num =  df_train[numeric_headers].to_numpy()\n",
    "X_test_num = df_test[numeric_headers].to_numpy()\n",
    "\n",
    "\n",
    "# we need to create separate lists for each branch\n",
    "crossed_outputs = []\n",
    "\n",
    "# CROSSED DATA INPUT\n",
    "input_crossed = Input(shape=(X_train_crossed.shape[1],), dtype='int64', name='wide_inputs')\n",
    "for idx,col in enumerate(cross_col_df_names):\n",
    "    \n",
    "    # track what the maximum integer value will be for this variable\n",
    "    # which is the same as the number of categories\n",
    "    N = max(df_train[col].max(),df_test[col].max())+1\n",
    "    N = len(encoders[col].classes_)\n",
    "    N_reduced = int(np.sqrt(N))\n",
    "    \n",
    "    \n",
    "    # this line of code does this: input_branch[:,idx]\n",
    "    x = tf.gather(input_crossed, idx, axis=1)\n",
    "    \n",
    "    # now use an embedding to deal with integers as if they were one hot encoded\n",
    "    x = Embedding(input_dim=N, \n",
    "                  output_dim=N_reduced, \n",
    "                  input_length=1, name=col+'_embed')(x)\n",
    "    \n",
    "    # save these outputs to concatenate later\n",
    "    crossed_outputs.append(x)\n",
    "    \n",
    "\n",
    "# now concatenate the outputs and add a fully connected layer\n",
    "wide_branch = concatenate(crossed_outputs, name='wide_concat')\n",
    "\n",
    "# reset this input branch\n",
    "all_deep_branch_outputs = []\n",
    "\n",
    "# CATEGORICAL DATA INPUT\n",
    "input_cat = Input(shape=(X_train_cat.shape[1],), dtype='int64', name='categorical_input')\n",
    "for idx,col in enumerate(categorical_headers_ints):\n",
    "    \n",
    "    # track what the maximum integer value will be for this variable\n",
    "    # which is the same as the number of categories\n",
    "    N = max(df_train[col].max(),df_test[col].max())+1\n",
    "    N_reduced = int(np.sqrt(N))\n",
    "    \n",
    "    # this line of code does this: input_branch[:,idx]\n",
    "    x = tf.gather(input_cat, idx, axis=1)\n",
    "    \n",
    "    # now use an embedding to deal with integers as if they were one hot encoded\n",
    "    x = Embedding(input_dim=N, \n",
    "                  output_dim=N_reduced, \n",
    "                  input_length=1, name=col+'_embed')(x)\n",
    "    \n",
    "    # save these outputs to concatenate later\n",
    "    all_deep_branch_outputs.append(x)\n",
    "    \n",
    "# NUMERIC DATA INPUT\n",
    "# create dense input branch for numeric\n",
    "input_num = Input(shape=(X_train_num.shape[1],), name='numeric')\n",
    "x_dense = Dense(units=20, activation='relu',name='num_1')(input_num)\n",
    "    \n",
    "all_deep_branch_outputs.append(x_dense)\n",
    "\n",
    "\n",
    "# merge the deep branches together\n",
    "deep_branch = concatenate(all_deep_branch_outputs,name='concat_embeds')\n",
    "deep_branch = Dense(units=50,activation='relu', name='deep1')(deep_branch)\n",
    "deep_branch = Dense(units=25,activation='relu', name='deep2')(deep_branch)\n",
    "deep_branch = Dense(units=10,activation='relu', name='deep3')(deep_branch)\n",
    "    \n",
    "# merge the deep and wide branch\n",
    "final_branch = concatenate([wide_branch, deep_branch],\n",
    "                           name='concat_deep_wide')\n",
    "final_branch = Dense(units=1,activation='sigmoid',\n",
    "                     name='combined')(final_branch)\n",
    "\n",
    "model = Model(inputs=[input_crossed,input_cat,input_num], \n",
    "              outputs=final_branch)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# you will need to install pydot properly on your machine to get this running\n",
    "plot_model(\n",
    "    model, to_file='model.png', show_shapes=True, show_layer_names=True,\n",
    "    rankdir='LR', expand_nested=False, dpi=96\n",
    ")\n",
    "# model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%time\n",
    "\n",
    "model.compile(optimizer='adagrad',\n",
    "              loss='mean_squared_error',\n",
    "              metrics=['accuracy'])\n",
    "\n",
    "# lets also add the history variable to see how we are doing\n",
    "# and lets add a validation set to keep track of our progress\n",
    "history = model.fit([X_train_crossed,X_train_cat,X_train_num],\n",
    "                    y_train, \n",
    "                    epochs=15, \n",
    "                    batch_size=32, \n",
    "                    verbose=1, \n",
    "                    validation_data = ([X_test_crossed,X_test_cat,X_test_num],y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "history = model.fit([X_train_crossed,X_train_cat,X_train_num],\n",
    "                    y_train, \n",
    "                    epochs=50, \n",
    "                    batch_size=32, \n",
    "                    verbose=0, \n",
    "                    validation_data = ([X_test_crossed,X_test_cat,X_test_num],y_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "yhat = np.round(model.predict([X_test_crossed,X_test_cat,X_test_num]))\n",
    "print(mt.confusion_matrix(y_test,yhat))\n",
    "print(mt.classification_report(y_test,yhat))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from matplotlib import pyplot as plt\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "plt.figure(figsize=(10,4))\n",
    "plt.subplot(2,2,1)\n",
    "plt.plot(history.history['accuracy'])\n",
    "\n",
    "plt.ylabel('Accuracy %')\n",
    "plt.title('Training')\n",
    "plt.subplot(2,2,2)\n",
    "plt.plot(history.history['val_accuracy'])\n",
    "plt.title('Validation')\n",
    "\n",
    "plt.subplot(2,2,3)\n",
    "plt.plot(history.history['loss'])\n",
    "plt.ylabel('Training Loss')\n",
    "plt.xlabel('epochs')\n",
    "\n",
    "plt.subplot(2,2,4)\n",
    "plt.plot(history.history['val_loss'])\n",
    "plt.xlabel('epochs')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Making Generalization Better\n",
    "In what ways might we try to make the model found generalize more concretely?\n",
    "\n",
    "**Self Test:** What is the best method to make the results better?  \n",
    "- A. Use early stopping\n",
    "- B. Use different optimization schemes\n",
    "- C. Perturb the Categorical Data during Training (augmentation)\n",
    "- D. Add L2 regularization to different layer's weights"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# In this Notebook You Learned\n",
    "- The keras API\n",
    "- Embedding layers in keras\n",
    "- Wide and Deep networks"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}