supervised/school-budgets.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# School Budgets DataCamp\n",
    "\n",
    "## EDA"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.inf()\n",
    "\n",
    "# Print the summary statistics\n",
    "print(df.describe())\n",
    "\n",
    "# Import matplotlib.pyplot as plt\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Create the histogram\n",
    "plt.hist(df['FTE'].dropna())\n",
    "\n",
    "# Add title and labels\n",
    "plt.title('Distribution of %full-time \\n employee works')\n",
    "plt.xlabel('% of full-time')\n",
    "plt.ylabel('num employees')\n",
    "\n",
    "# Display the histogram\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Exploring datatypes in pandas\n",
    "It's always good to know what datatypes you're working with, especially when the inefficient pandas type object may be involved. Towards that end, let's explore what we have.\n",
    "\n",
    "The data has been loaded into the workspace as df. Your job is to look at the DataFrame attribute .dtypes in the IPython Shell, and call its .value_counts() method in order to answer the question below.\n",
    "\n",
    "Make sure to call df.dtypes.value_counts(), and not df.value_counts()! Check out the difference in the Shell. df.value_counts() will return an error, because it is a Series method, not a DataFrame method.\n",
    "\n",
    "### Encode the labels as categorical variables\n",
    "Remember, your ultimate goal is to predict the probability that a certain label is attached to a budget line item. You just saw that many columns in your data are the inefficient object type. Does this include the labels you're trying to predict? Let's find out!\n",
    "\n",
    "There are 9 columns of labels in the dataset. Each of these columns is a category that has many possible values it can take). The 9 labels have been loaded into a list called LABELS. In the Shell, check out the type for these labels using df[LABELS].dtypes.\n",
    "\n",
    "You will notice that every label is encoded as an object datatype. Because category datatypes are much more efficient your task is to convert the labels to category types using the .astype() method.\n",
    "\n",
    "Note: .astype() only works on a pandas Series. Since you are working with a pandas DataFrame, you'll need to use the .apply() method and provide a lambda function called categorize_label that applies .astype() to each column, x.\n",
    "\n",
    "### Counting unique labels\n",
    "As Peter mentioned in the video, there are over 100 unique labels. In this exercise, you will explore this fact by counting and plotting the number of unique values for each category of label.\n",
    "\n",
    "The dataframe df and the LABELS list have been loaded into the workspace; the LABELS columns of df have been converted to category types.\n",
    "\n",
    "pandas, which has been pre-imported as pd, provides a pd.Series.nunique method for counting the number of unique values in a Series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.dtpyes.value_counts()\n",
    "\n",
    "# Define the lambda function: categorize_label\n",
    "categorize_label = lambda x: x.astype('category')\n",
    "\n",
    "# Convert df[LABELS] to a categorical type\n",
    "df[LABELS] = df[LABELS].apply(categorize_label, axis=0)\n",
    "\n",
    "# Print the converted dtypes\n",
    "print(df[LABELS].dtypes)\n",
    "\n",
    "# Import matplotlib.pyplot\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Calculate number of unique values for each label: num_unique_labels\n",
    "num_unique_labels = df[LABELS].apply(pd.Series.nunique, axis=0)\n",
    "\n",
    "# Plot number of unique values for each label\n",
    "num_unique_labels.plot(kind='bar')\n",
    "\n",
    "# Label the axes\n",
    "plt.xlabel('Labels')\n",
    "plt.ylabel('Number of unique values')\n",
    "\n",
    "# Display the plot\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Computing log loss with NumPy\n",
    "\n",
    "Log loss penalizes highly confident wrong answers much more than any other type.\n",
    "\n",
    "To see how the log loss metric handles the trade-off between accuracy and confidence, we will use some sample data generated with NumPy and compute the log loss using the provided function compute_log_loss(), which Peter showed you in the video.\n",
    "\n",
    "5 one-dimensional numeric arrays simulating different types of predictions have been pre-loaded: actual_labels, correct_confident, correct_not_confident, wrong_not_confident, and wrong_confident.\n",
    "\n",
    "Your job is to compute the log loss for each sample set provided using the compute_log_loss(predicted_values, actual_values). It takes the predicted values as the first argument and the actual values as the second argument."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Compute and print log loss for 1st case\n",
    "correct_confident = compute_log_loss(correct_confident, actual_labels)\n",
    "print(\"Log loss, correct and confident: {}\".format(correct_confident)) \n",
    "\n",
    "# Compute log loss for 2nd case\n",
    "correct_not_confident = compute_log_loss(correct_not_confident, actual_labels)\n",
    "print(\"Log loss, correct and not confident: {}\".format(correct_not_confident)) \n",
    "\n",
    "# Compute and print log loss for 3rd case\n",
    "wrong_not_confident = compute_log_loss(wrong_not_confident, actual_labels)\n",
    "print(\"Log loss, wrong and not confident: {}\".format(wrong_not_confident)) \n",
    "\n",
    "# Compute and print log loss for 4th case\n",
    "wrong_confident = compute_log_loss(wrong_confident, actual_labels)\n",
    "print(\"Log loss, wrong and confident: {}\".format(wrong_confident)) \n",
    "\n",
    "# Compute and print log loss for actual labels\n",
    "actual_labels = compute_log_loss(actual_labels, actual_labels)\n",
    "print(\"Log loss, actual labels: {}\".format(actual_labels)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Models\n",
    "\n",
    "### Simple models\n",
    "- Start with a simple model first\n",
    "- Lets you understand how challenging a problem is\n",
    "- How much signal can we get with basic methods?\n",
    "- Can start with model that just uses numeric columns\n",
    "- Good place to start: multi-class logistic regression\n",
    "    - Treat each column as independent\n",
    "    - Train classifer on each label and use these to predict\n",
    "\n",
    "### Splitting data\n",
    "- train-test split will not work as there are a lot of labels and the dataset is quite small\n",
    "- Could use stratified StratifiedShuffleSplit, but this only works with a single target variable\n",
    "- Instead for a multilabel problem you'll need to develop a multilabel train-test split method (provided by datacamp)\n",
    "\n",
    "### Missing values\n",
    "- Set to -1000 so its treated differently to 0\n",
    "\n",
    "### OneVsRestClassifier\n",
    "- sklearn classifier treats each column of y independently\n",
    "- Fits seperate classifier to each of the columns\n",
    "\n",
    "#### Setting up a train-test split in scikit-learn\n",
    "Alright, you've been patient and awesome. It's finally time to start training models!\n",
    "\n",
    "The first step is to split the data into a training set and a test set. Some labels don't occur very often, but we want to make sure that they appear in both the training and the test sets. We provide a function that will make sure at least min_count examples of each label appear in each split: multilabel_train_test_split.\n",
    "\n",
    "Feel free to check out the full code for multilabel_train_test_split here.\n",
    "\n",
    "You'll start with a simple model that uses just the numeric columns of your DataFrame when calling multilabel_train_test_split. The data has been read into a DataFrame df and a list consisting of just the numeric columns is available as NUMERIC_COLUMNS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from warnings import warn\n",
    "\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "def multilabel_sample(y, size=1000, min_count=5, seed=None):\n",
    "    \"\"\" Takes a matrix of binary labels `y` and returns\n",
    "        the indices for a sample of size `size` if\n",
    "        `size` > 1 or `size` * len(y) if size =< 1.\n",
    "        The sample is guaranteed to have > `min_count` of\n",
    "        each label.\n",
    "    \"\"\"\n",
    "    try:\n",
    "        if (np.unique(y).astype(int) != np.array([0, 1])).all():\n",
    "            raise ValueError()\n",
    "    except (TypeError, ValueError):\n",
    "        raise ValueError('multilabel_sample only works with binary indicator matrices')\n",
    "\n",
    "    if (y.sum(axis=0) < min_count).any():\n",
    "        raise ValueError('Some classes do not have enough examples. Change min_count if necessary.')\n",
    "\n",
    "    if size <= 1:\n",
    "        size = np.floor(y.shape[0] * size)\n",
    "\n",
    "    if y.shape[1] * min_count > size:\n",
    "        msg = \"Size less than number of columns * min_count, returning {} items instead of {}.\"\n",
    "        warn(msg.format(y.shape[1] * min_count, size))\n",
    "        size = y.shape[1] * min_count\n",
    "\n",
    "    rng = np.random.RandomState(seed if seed is not None else np.random.randint(1))\n",
    "\n",
    "    if isinstance(y, pd.DataFrame):\n",
    "        choices = y.index\n",
    "        y = y.values\n",
    "    else:\n",
    "        choices = np.arange(y.shape[0])\n",
    "\n",
    "    sample_idxs = np.array([], dtype=choices.dtype)\n",
    "\n",
    "    # first, guarantee > min_count of each label\n",
    "    for j in range(y.shape[1]):\n",
    "        label_choices = choices[y[:, j] == 1]\n",
    "        label_idxs_sampled = rng.choice(label_choices, size=min_count, replace=False)\n",
    "        sample_idxs = np.concatenate([label_idxs_sampled, sample_idxs])\n",
    "\n",
    "    sample_idxs = np.unique(sample_idxs)\n",
    "\n",
    "    # now that we have at least min_count of each, we can just random sample\n",
    "    sample_count = int(size - sample_idxs.shape[0])\n",
    "\n",
    "    # get sample_count indices from remaining choices\n",
    "    remaining_choices = np.setdiff1d(choices, sample_idxs)\n",
    "    remaining_sampled = rng.choice(remaining_choices,\n",
    "                                   size=sample_count,\n",
    "                                   replace=False)\n",
    "\n",
    "    return np.concatenate([sample_idxs, remaining_sampled])\n",
    "\n",
    "\n",
    "def multilabel_sample_dataframe(df, labels, size, min_count=5, seed=None):\n",
    "    \"\"\" Takes a dataframe `df` and returns a sample of size `size` where all\n",
    "        classes in the binary matrix `labels` are represented at\n",
    "        least `min_count` times.\n",
    "    \"\"\"\n",
    "    idxs = multilabel_sample(labels, size=size, min_count=min_count, seed=seed)\n",
    "    return df.loc[idxs]\n",
    "\n",
    "\n",
    "def multilabel_train_test_split(X, Y, size, min_count=5, seed=None):\n",
    "    \"\"\" Takes a features matrix `X` and a label matrix `Y` and\n",
    "        returns (X_train, X_test, Y_train, Y_test) where all\n",
    "        classes in Y are represented at least `min_count` times.\n",
    "    \"\"\"\n",
    "    index = Y.index if isinstance(Y, pd.DataFrame) else np.arange(Y.shape[0])\n",
    "\n",
    "    test_set_idxs = multilabel_sample(Y, size=size, min_count=min_count, seed=seed)\n",
    "    train_set_idxs = np.setdiff1d(index, test_set_idxs)\n",
    "\n",
    "    test_set_mask = index.isin(test_set_idxs)\n",
    "    train_set_mask = ~test_set_mask\n",
    "\n",
    "    return (X[train_set_mask], X[test_set_mask], Y[train_set_mask], Y[test_set_mask])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the new DataFrame: numeric_data_only\n",
    "numeric_data_only = df[NUMERIC_COLUMNS].fillna(-1000)\n",
    "\n",
    "# Get labels and convert to dummy variables: label_dummies\n",
    "label_dummies = pd.get_dummies(df[LABELS])\n",
    "\n",
    "# Create training and test sets\n",
    "X_train, X_test, y_train, y_test = multilabel_train_test_split(numeric_data_only,\n",
    "                                                               label_dummies,\n",
    "                                                               size=0.2, \n",
    "                                                               seed=123)\n",
    "\n",
    "# Print the info\n",
    "print(\"X_train info:\")\n",
    "print(X_train.info())\n",
    "print(\"\\nX_test info:\")  \n",
    "print(X_test.info())\n",
    "print(\"\\ny_train info:\")  \n",
    "print(y_train.info())\n",
    "print(\"\\ny_test info:\")  \n",
    "print(y_test.info()) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Training a model\n",
    "With split data in hand, you're only a few lines away from training a model.\n",
    "\n",
    "In this exercise, you will import the logistic regression and one versus rest classifiers in order to fit a multi-class logistic regression model to the NUMERIC_COLUMNS of your feature data.\n",
    "\n",
    "Then you'll test and print the accuracy with the .score() method to see the results of training.\n",
    "\n",
    "Before you train! Remember, we're ultimately going to be using logloss to score our model, so don't worry too much about the accuracy here. Keep in mind that you're throwing away all of the text data in the dataset - that's by far most of the data! So don't get your hopes up for a killer performance just yet. We're just interested in getting things up and running at the moment.\n",
    "\n",
    "All data necessary to call multilabel_train_test_split() has been loaded into the workspace."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import classifiers\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.multiclass import OneVsRestClassifier\n",
    "\n",
    "# Create the DataFrame: numeric_data_only\n",
    "numeric_data_only = df[NUMERIC_COLUMNS].fillna(-1000)\n",
    "\n",
    "# Get labels and convert to dummy variables: label_dummies\n",
    "label_dummies = pd.get_dummies(df[LABELS])\n",
    "\n",
    "# Create training and test sets\n",
    "X_train, X_test, y_train, y_test = multilabel_train_test_split(numeric_data_only,\n",
    "                                                               label_dummies,\n",
    "                                                               size=0.2, \n",
    "                                                               seed=123)\n",
    "\n",
    "# Instantiate the classifier: clf\n",
    "clf = OneVsRestClassifier(LogisticRegression())\n",
    "\n",
    "# Fit the classifier to the training data\n",
    "clf.fit(X_train, y_train)\n",
    "\n",
    "# Print the accuracy\n",
    "print(\"Accuracy: {}\".format(clf.score(X_test, y_test)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Use your model to predict values on holdout data\n",
    "You're ready to make some predictions! Remember, the train-test-split you've carried out so far is for model development. The original competition provides an additional test set, for which you'll never actually see the correct labels. This is called the \"holdout data.\"\n",
    "\n",
    "The point of the holdout data is to provide a fair test for machine learning competitions. If the labels aren't known by anyone but DataCamp, DrivenData, or whoever is hosting the competition, you can be sure that no one submits a mere copy of labels to artificially pump up the performance on their model.\n",
    "\n",
    "Remember that the original goal is to predict the probability of each label. In this exercise you'll do just that by using the .predict_proba() method on your trained model.\n",
    "\n",
    "First, however, you'll need to load the holdout data, which is available in the workspace as the file HoldoutData.csv."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Instantiate the classifier: clf\n",
    "clf = OneVsRestClassifier(LogisticRegression())\n",
    "\n",
    "# Fit it to the training data\n",
    "clf.fit(X_train, y_train)\n",
    "\n",
    "# Load the holdout data: holdout\n",
    "holdout = pd.read_csv('HoldoutData.csv', index_col=0)\n",
    "\n",
    "# Generate predictions: predictions\n",
    "predictions = clf.predict_proba(holdout[NUMERIC_COLUMNS].fillna(-1000))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a bag-of-words in scikit-learn\n",
    "In this exercise, you'll study the effects of tokenizing in different ways by comparing the bag-of-words representations resulting from different token patterns.\n",
    "\n",
    "You will focus on one feature only, the Position_Extra column, which describes any additional information not captured by the Position_Type label.\n",
    "\n",
    "For example, in the Shell you can check out the budget item in row 8960 of the data using df.loc[8960]. Looking at the output reveals that this Object_Description is overtime pay. For who? The Position Type is merely \"other\", but the Position Extra elaborates: \"BUS DRIVER\". Explore the column further to see more instances. It has a lot of NaN values.\n",
    "\n",
    "Your task is to turn the raw text in this column into a bag-of-words representation by creating tokes that contain only alphanumeric characters.\n",
    "\n",
    "For comparison purposes, the first 15 tokens of vec_basic, which splits df.Position_Extra into tokens when it encounters only whitespace characters, have been printed along with the length of the representation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}