evergreen_benchmark.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Stumbling Upon Evergreens, Our Kaggle Competition\n",
    "\n",
    "Let's get started with a benchmark for our new Kaggle competition! Kaggle and StumbleUpon have allowed us to bring back the pair's previous competition. Stumble Upon is a \"user-curated web content discovery engine that recommends relevant, high quality pages and media to its users, based on their interests\". The challenge is present their users with just the right information at the right time. Timing is crucial here. No one wants to be recommended holiday recipes in the middle of the summer. Here's the game: can we build a model that will predict whether or not a given recommendation will have enduring appeal? Stumble Upon calls this enduring content 'evergreen'. All other content with short life spans are called 'ephemeral'. Can you predict what content will stand the test of time?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn import metrics, cross_validation\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.preprocessing import StandardScaler, MinMaxScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "def cv_loop(X, y, model, K):\n",
    "    '''\n",
    "    Cross validation: for K iterations, split the data into train and test\n",
    "    sets, build a model, and return the mean AUC.\n",
    "    '''\n",
    "    \n",
    "    SEED = 15\n",
    "    mean_auc = 0.\n",
    "    for i in range(K):\n",
    "        X_train, X_cv, y_train, y_cv = cross_validation.train_test_split(\n",
    "            X, y, test_size = 0.2,\n",
    "            random_state = i*SEED)\n",
    "        model.fit(X_train, y_train)\n",
    "        preds = model.predict_proba(X_cv)[:,1]\n",
    "        auc = metrics.roc_auc_score(y_cv, preds)\n",
    "        print(\"AUC (fold %d/%d): %f\" % (i + 1, K, auc))\n",
    "        mean_auc += auc\n",
    "    return mean_auc/K\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading in the Data\n",
    "\n",
    "First let's read in the data downloaded from the Kaggle competition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Unnamed: 0</th>\n",
       "      <th>url</th>\n",
       "      <th>urlid</th>\n",
       "      <th>boilerplate</th>\n",
       "      <th>alchemy_category</th>\n",
       "      <th>alchemy_category_score</th>\n",
       "      <th>avglinksize</th>\n",
       "      <th>commonlinkratio_1</th>\n",
       "      <th>commonlinkratio_2</th>\n",
       "      <th>commonlinkratio_3</th>\n",
       "      <th>...</th>\n",
       "      <th>is_news</th>\n",
       "      <th>lengthyLinkDomain</th>\n",
       "      <th>linkwordscore</th>\n",
       "      <th>news_front_page</th>\n",
       "      <th>non_markup_alphanum_characters</th>\n",
       "      <th>numberOfLinks</th>\n",
       "      <th>numwords_in_url</th>\n",
       "      <th>parametrizedLinkRatio</th>\n",
       "      <th>spelling_errors_ratio</th>\n",
       "      <th>label</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2351</td>\n",
       "      <td>http://ciaobella50.com/?p=4679</td>\n",
       "      <td>2407</td>\n",
       "      <td>{\"title\":\"Funky Tie Dyed and Fabulous \",\"body\"...</td>\n",
       "      <td>arts_entertainment</td>\n",
       "      <td>0.608835</td>\n",
       "      <td>1.380000</td>\n",
       "      <td>0.430000</td>\n",
       "      <td>0.190000</td>\n",
       "      <td>0.040000</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>28</td>\n",
       "      <td>1</td>\n",
       "      <td>1511</td>\n",
       "      <td>100</td>\n",
       "      <td>0</td>\n",
       "      <td>0.810000</td>\n",
       "      <td>0.124031</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1115</td>\n",
       "      <td>http://freefashioninternships.com/</td>\n",
       "      <td>2605</td>\n",
       "      <td>{\"title\":\"Free Fashion Internships com Fashion...</td>\n",
       "      <td>business</td>\n",
       "      <td>0.827129</td>\n",
       "      <td>2.558824</td>\n",
       "      <td>0.529412</td>\n",
       "      <td>0.352941</td>\n",
       "      <td>0.029412</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>1</td>\n",
       "      <td>4448</td>\n",
       "      <td>34</td>\n",
       "      <td>0</td>\n",
       "      <td>0.441176</td>\n",
       "      <td>0.115254</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1049</td>\n",
       "      <td>http://www.manjulaskitchen.com/</td>\n",
       "      <td>1202</td>\n",
       "      <td>{\"title\":\"Manjula s Kitchen Indian Vegetarian ...</td>\n",
       "      <td>business</td>\n",
       "      <td>0.768478</td>\n",
       "      <td>1.210145</td>\n",
       "      <td>0.401408</td>\n",
       "      <td>0.070423</td>\n",
       "      <td>0.007042</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>47</td>\n",
       "      <td>1</td>\n",
       "      <td>864</td>\n",
       "      <td>142</td>\n",
       "      <td>0</td>\n",
       "      <td>0.021127</td>\n",
       "      <td>0.041667</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2790</td>\n",
       "      <td>http://ohsheglows.com/recipage?recipe_id=6002030</td>\n",
       "      <td>6418</td>\n",
       "      <td>{\"title\":\"Recipage Oh She Glows \",\"body\":\"I ve...</td>\n",
       "      <td>?</td>\n",
       "      <td>?</td>\n",
       "      <td>1.350000</td>\n",
       "      <td>0.608696</td>\n",
       "      <td>0.103261</td>\n",
       "      <td>0.032609</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>0</td>\n",
       "      <td>13117</td>\n",
       "      <td>184</td>\n",
       "      <td>2</td>\n",
       "      <td>0.326087</td>\n",
       "      <td>0.101562</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2829</td>\n",
       "      <td>http://www.perpetualkid.com/index.asp?PageActi...</td>\n",
       "      <td>1159</td>\n",
       "      <td>{\"title\":\"HEART SWALLOW BUSINESS CARD HOLDER \"...</td>\n",
       "      <td>business</td>\n",
       "      <td>0.735727</td>\n",
       "      <td>1.909091</td>\n",
       "      <td>0.533333</td>\n",
       "      <td>0.238095</td>\n",
       "      <td>0.047619</td>\n",
       "      <td>...</td>\n",
       "      <td>?</td>\n",
       "      <td>0</td>\n",
       "      <td>63</td>\n",
       "      <td>?</td>\n",
       "      <td>530</td>\n",
       "      <td>105</td>\n",
       "      <td>6</td>\n",
       "      <td>0.133333</td>\n",
       "      <td>0.089286</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 28 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Unnamed: 0                                                url  urlid  \\\n",
       "0        2351                     http://ciaobella50.com/?p=4679   2407   \n",
       "1        1115                 http://freefashioninternships.com/   2605   \n",
       "2        1049                    http://www.manjulaskitchen.com/   1202   \n",
       "3        2790   http://ohsheglows.com/recipage?recipe_id=6002030   6418   \n",
       "4        2829  http://www.perpetualkid.com/index.asp?PageActi...   1159   \n",
       "\n",
       "                                         boilerplate    alchemy_category  \\\n",
       "0  {\"title\":\"Funky Tie Dyed and Fabulous \",\"body\"...  arts_entertainment   \n",
       "1  {\"title\":\"Free Fashion Internships com Fashion...            business   \n",
       "2  {\"title\":\"Manjula s Kitchen Indian Vegetarian ...            business   \n",
       "3  {\"title\":\"Recipage Oh She Glows \",\"body\":\"I ve...                   ?   \n",
       "4  {\"title\":\"HEART SWALLOW BUSINESS CARD HOLDER \"...            business   \n",
       "\n",
       "  alchemy_category_score  avglinksize  commonlinkratio_1  commonlinkratio_2  \\\n",
       "0               0.608835     1.380000           0.430000           0.190000   \n",
       "1               0.827129     2.558824           0.529412           0.352941   \n",
       "2               0.768478     1.210145           0.401408           0.070423   \n",
       "3                      ?     1.350000           0.608696           0.103261   \n",
       "4               0.735727     1.909091           0.533333           0.238095   \n",
       "\n",
       "   commonlinkratio_3  ...    is_news  lengthyLinkDomain  linkwordscore  \\\n",
       "0           0.040000  ...          1                  0             28   \n",
       "1           0.029412  ...          1                  0              9   \n",
       "2           0.007042  ...          1                  0             47   \n",
       "3           0.032609  ...          1                  1              8   \n",
       "4           0.047619  ...          ?                  0             63   \n",
       "\n",
       "   news_front_page  non_markup_alphanum_characters  numberOfLinks  \\\n",
       "0                1                            1511            100   \n",
       "1                1                            4448             34   \n",
       "2                1                             864            142   \n",
       "3                0                           13117            184   \n",
       "4                ?                             530            105   \n",
       "\n",
       "   numwords_in_url  parametrizedLinkRatio spelling_errors_ratio  label  \n",
       "0                0               0.810000              0.124031      0  \n",
       "1                0               0.441176              0.115254      0  \n",
       "2                0               0.021127              0.041667      1  \n",
       "3                2               0.326087              0.101562      1  \n",
       "4                6               0.133333              0.089286      0  \n",
       "\n",
       "[5 rows x 28 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train = pd.read_csv('/Users/youngtodd/Documents/utkml_examples/stumble_evergreen/data/train.csv')\n",
    "test = pd.read_csv('/Users/youngtodd/Documents/utkml_examples/stumble_evergreen/data/test.csv')\n",
    "\n",
    "train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: there is some missing data within the training and test sets encoded with '?'. Let's take the naive approach and set all missing data to 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train = train.replace('?', value=0)\n",
    "test = test.replace('?', value=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Starting Off\n",
    "\n",
    "Some of the best information in this dataset is contained in some natural language features e.g. 'boilerplate', the description of the website to be promoted. While we ultimately want to make use of this information, modeling it can be a bit tricky. Here we will first take a look at what we can do with only the numeric features of the data. This will serve as a benchmark for all of our future modeling. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Our first model will only make use of numeric features\n",
    "# Let's also separate the label from the training features\n",
    "X_train = train[train.columns[5:-1]]\n",
    "y_train = train['label']\n",
    "\n",
    "# Note: the test set does not have a label (we need to predict this)\n",
    "X_test = test[test.columns[5:]]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocesssing\n",
    "\n",
    "Many of the models work better when the data is represented on a common scale. This allows us a sense of magnitude between the features we want to learn from."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "X_train = StandardScaler().fit_transform(X_train)\n",
    "X_test = StandardScaler().fit_transform(X_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### On With It, Train The Model!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "AUC score for 10 fold cross validation:\n",
      "\n",
      "AUC (fold 1/10): 0.604088\n",
      "AUC (fold 2/10): 0.600203\n",
      "AUC (fold 3/10): 0.613043\n",
      "AUC (fold 4/10): 0.653669\n",
      "AUC (fold 5/10): 0.637583\n",
      "AUC (fold 6/10): 0.632930\n",
      "AUC (fold 7/10): 0.631132\n",
      "AUC (fold 8/10): 0.611367\n",
      "AUC (fold 9/10): 0.622946\n",
      "AUC (fold 10/10): 0.614334\n",
      "\n",
      "Mean AUC: 0.622129.2\n"
     ]
    }
   ],
   "source": [
    "# Gaussian NB for real valued features.\n",
    "model = GaussianNB()\n",
    "K = 10 # Number of iterations for cross validation\n",
    "\n",
    "print(\"AUC score for 10 fold cross validation:\\n\")\n",
    "score = cv_loop(X_train, y_train, model, K)\n",
    "\n",
    "print(\"\\nMean AUC: %f\" %score)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### First thoughts\n",
    "\n",
    "That is not too bad for an incredibly quick model. Assuming that classes are relatively balanced, our model is approximately 12% better than guessing randomly. With some work, we should be able to improve that score. For now, let's formally fit this model using all the training data (above we were training on smaller samples of the dataset)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Fitting the Gaussian NB\n",
    "model.fit(X_train, y_train)\n",
    "\n",
    "# Let's make predictions on the unseen test set\n",
    "preds = model.predict_proba(X_test)[:,1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Kaggle Submission\n",
    "\n",
    "Our submissions to the Kaggle competition are in the form of .csv files with two columns: 1. the URL ID feature of the test set; 2. our predictions for that URL ID (these will be real valued estimates for the probability that our URL is an 'evergreen' [label=1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Get the url ID from the test set\n",
    "urlid = test['urlid']\n",
    "\n",
    "# combine the predictions and url IDs into a pandas dataframe\n",
    "# Note: this is easy with Python's dictionaries, but make sure to set\n",
    "# the index to url ID (otherwise the column orders will be random. Reason=dictionaries are unordered)\n",
    "pred_df = pd.DataFrame({'label':preds, 'urlid': urlid}).set_index('urlid')\n",
    "pred_df.to_csv('first_evergreen_submission')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python [conda root]",
   "language": "python",
   "name": "conda-root-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}