tutorials/Basics.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basics Tutorial for Snorkel MeTaL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The purpose of this tutorial is to introduce the basic pipeline, classes, and debugging/analysis tools in Snorkel MeTaL. In this notebook, we'll investigate a single-task problem with synthetic data, with an emphasis on the basic design concepts of MeTaL. Check out the `tutorials/` directory to see how this basic pipeline extends to the full _multi-task_ setting, and other topics.\n",
    "\n",
    "This tutorial consists of four steps, following the basic [data programming](https://arxiv.org/abs/1605.07723) pipeline as in [Snorkel](https://hazyresearch.github.io/snorkel/):\n",
    "1. **Load Data:** In the _weakly supervised_ setting, we only have access to unlabeled data points `X`, matrix of noisy labels `L`, and dev/test labels `Y`*\n",
    "2. **Train Label Model:** The purpose of the `LabelModel` is to estimate the unknown accuracies of the labeling functions, _without access to `Y`_, and then use this to denoise and combine them into a set of _probabilistic training labels_.\n",
    "3. **Train End Model:** We can then use these training labels to supervise a discriminative classifier!\n",
    "4. **Evaluate:** We evaluate this model on a held-out test set\n",
    "\n",
    "Let's get started!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We first need to make sure that the `metal/` directory is on our Python path. If the following cell runs without an error, you're all set. If not, make sure that you've installed `snorkel-metal` with pip (conda coming soon) or that you've added the repo to your path if you're running from source; for example, running `source add_to_path.sh` from the repository root."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append('../../metal')\n",
    "import metal"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Load Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step in a Snorkel MeTaL application is preparing your data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Concept 1: Required Data Types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In particular, MeTaL makes use of the following types of data (n = # examples, m = # labeling functions):\n",
    "* X: an \\[n\\]-dim iterable of end model inputs (e.g., feature vectors or encoded sentences for an RNN)\n",
    "* Y: an \\[n\\]-dim numpy.ndarray of target labels ($Y \\in \\{1,...,k\\}^n$)\n",
    "* L: an \\[n,m\\] scipy.sparse matrix of noisy labels ($L \\in \\{0,...,k\\}^{n \\times m}$, with label 0 reserved for abstentions\n",
    "\n",
    "And optionally (for use with some debugging/analysis tools):\n",
    "* D: an \\[n\\]-dim iterable of human-readable examples (e.g., sentences)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we use pre-generated synthetic data, where each example is a bag of words drawn from a different gaussian distribution for each class from a 1000-word vocabulary, and our features are simply the 1000-dimensional vector of counts for each word. For some excellent resources on how to write labeling functions, see the Snorkel tutorials at https://github.com/HazyResearch/snorkel/tree/master/tutorials."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "with open(\"data/basics_tutorial.pkl\", 'rb') as f:\n",
    "    X, Y, L, D = pickle.load(f)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([10000, 1000])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(10000,)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "text/plain": [
       "(10000, 10)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.shape\n",
    "Y.shape\n",
    "L.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you need to divide your data into splits, you can do so with the provided utility function. We split our data 80/10/10 into train/dev/test, stratifying by the labels in `Y` to ensure a similar class balance in each split."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "from metal.utils import split_data\n",
    "\n",
    "Xs, Ys, Ls, Ds = split_data(X, Y, L, D, splits=[0.8, 0.1, 0.1], stratify_by=Y, seed=123)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Investigate Label Matrices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "MeTaL comes with a number of analysis tools for investigating your label matrix. \n",
    "For example, you can view summary statistics for each labeling function.\n",
    "If you have a dev set with gold labels, you can view the empirical accuracy of your functions, in addition to the core summary statistics. If you do not have a dev set with gold labels, simply do not specify optional argument Y.\n",
    "\n",
    "Polarity is a list of classes that a labeling function (LF) predicts for one or more examples. Coverage is the fraction of examples that the labeling function predicts over (as opposed to labeling 0 to abstain). Overlaps is the fraction of examples that the labeling function predicts over that at least one other LF predicts over as well. Conflicts is the fraction of examples that the LF predicts as one class but at least one other LF predicts that same example as a different class. Correct, Incorrect, and Empirical Accuracy are all comparisons to the gold standard labels and thus omitted if Y is not specified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Polarity</th>\n",
       "      <th>Coverage</th>\n",
       "      <th>Overlaps</th>\n",
       "      <th>Conflicts</th>\n",
       "      <th>Correct</th>\n",
       "      <th>Incorrect</th>\n",
       "      <th>Emp. Acc.</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>[1, 2]</td>\n",
       "      <td>0.700</td>\n",
       "      <td>0.700</td>\n",
       "      <td>0.655</td>\n",
       "      <td>374</td>\n",
       "      <td>326</td>\n",
       "      <td>0.534286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>[1, 2]</td>\n",
       "      <td>0.735</td>\n",
       "      <td>0.735</td>\n",
       "      <td>0.700</td>\n",
       "      <td>468</td>\n",
       "      <td>267</td>\n",
       "      <td>0.636735</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>[1, 2]</td>\n",
       "      <td>0.744</td>\n",
       "      <td>0.744</td>\n",
       "      <td>0.701</td>\n",
       "      <td>500</td>\n",
       "      <td>244</td>\n",
       "      <td>0.672043</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>[1, 2]</td>\n",
       "      <td>0.748</td>\n",
       "      <td>0.748</td>\n",
       "      <td>0.706</td>\n",
       "      <td>540</td>\n",
       "      <td>208</td>\n",
       "      <td>0.721925</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>[1, 2]</td>\n",
       "      <td>0.719</td>\n",
       "      <td>0.719</td>\n",
       "      <td>0.675</td>\n",
       "      <td>454</td>\n",
       "      <td>265</td>\n",
       "      <td>0.631433</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>[1, 2]</td>\n",
       "      <td>0.746</td>\n",
       "      <td>0.746</td>\n",
       "      <td>0.696</td>\n",
       "      <td>457</td>\n",
       "      <td>289</td>\n",
       "      <td>0.612601</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>[1, 2]</td>\n",
       "      <td>0.762</td>\n",
       "      <td>0.762</td>\n",
       "      <td>0.728</td>\n",
       "      <td>509</td>\n",
       "      <td>253</td>\n",
       "      <td>0.667979</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>[1, 2]</td>\n",
       "      <td>0.796</td>\n",
       "      <td>0.796</td>\n",
       "      <td>0.744</td>\n",
       "      <td>593</td>\n",
       "      <td>203</td>\n",
       "      <td>0.744975</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>[1, 2]</td>\n",
       "      <td>0.811</td>\n",
       "      <td>0.811</td>\n",
       "      <td>0.756</td>\n",
       "      <td>642</td>\n",
       "      <td>169</td>\n",
       "      <td>0.791615</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>[1, 2]</td>\n",
       "      <td>0.784</td>\n",
       "      <td>0.784</td>\n",
       "      <td>0.735</td>\n",
       "      <td>559</td>\n",
       "      <td>225</td>\n",
       "      <td>0.713010</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Polarity  Coverage  Overlaps  Conflicts  Correct  Incorrect  Emp. Acc.\n",
       "0   [1, 2]     0.700     0.700      0.655      374        326   0.534286\n",
       "1   [1, 2]     0.735     0.735      0.700      468        267   0.636735\n",
       "2   [1, 2]     0.744     0.744      0.701      500        244   0.672043\n",
       "3   [1, 2]     0.748     0.748      0.706      540        208   0.721925\n",
       "4   [1, 2]     0.719     0.719      0.675      454        265   0.631433\n",
       "5   [1, 2]     0.746     0.746      0.696      457        289   0.612601\n",
       "6   [1, 2]     0.762     0.762      0.728      509        253   0.667979\n",
       "7   [1, 2]     0.796     0.796      0.744      593        203   0.744975\n",
       "8   [1, 2]     0.811     0.811      0.756      642        169   0.791615\n",
       "9   [1, 2]     0.784     0.784      0.735      559        225   0.713010"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from metal.analysis import lf_summary\n",
    "\n",
    "lf_summary(Ls[1],Y=Ys[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you're interested in more graphical investigative tools, you can take a look at the Visualization tutorial.  \n",
    "Once you're satisfied with your label matrices, it's time to aggregate the labels using the label model and train an end model with the resulting probabilistic labels."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Train Label Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Concept: Config dicts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All Snorkel MeTaL models are driven by configuration dictionaries, or config dicts; that is, all settings for running the given model are stored in a single dictionary, `self.config`. \n",
    "\n",
    "When a model is initialized or the train_model() method is called, any extra keyword arguments passed by the user will be used to update the config dict. Then, throughout the class's methods, whenever a setting needs to be looked up, it is pulled from this dictionary, rather than required as a keyword. This has a number of benefits:\n",
    "\n",
    "1. Default values are stored in only one place, the default config dict, rather than in multiple method signatures.\n",
    "1. Code maintenance is simplified, since fewer keyword arguments need to be passed up and down the call stack.\n",
    "2. Logging is simplified, since the complete settings for a given model can be logged by simply writing the config dict (a python dictionary) to file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Label Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When constructing a LabelModel, the only required argument is `k`, the cardinality of the task. For example, k=2 means a binary classification task where 1 and 2 are the class labels, and 0 is reserved for an abstain."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from metal.label_model import LabelModel\n",
    "label_model = LabelModel(k=2, seed=123)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The only required argument to `LabelModel.train_model()` is a label matrix. All other keyword arguments are optional. \n",
    "\n",
    "For example, if you know the class_balance of your data, you can pass that in as a list or array; alternatively, you may pass in the target labels of your dev set (`Y_dev`) and have the class balance estimated from that. If neither of these is provided, the classes are assumed to be uniformly distributed.\n",
    "\n",
    "Any other keyword arguments (e.g., the number of epochs (`n_epochs`), logging frequency (`log_train_every`), learning rate (`lr`), L2 regularization (`l2`), optimizer type (`optimizer`), etc.) will be used to update the config dict."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing O...\n",
      "Estimating \\mu...\n",
      "[50 epo]: TRAIN:[loss=0.039]\n",
      "[100 epo]: TRAIN:[loss=0.003]\n",
      "[150 epo]: TRAIN:[loss=0.002]\n",
      "[200 epo]: TRAIN:[loss=0.002]\n",
      "[250 epo]: TRAIN:[loss=0.002]\n",
      "[300 epo]: TRAIN:[loss=0.002]\n",
      "[350 epo]: TRAIN:[loss=0.002]\n",
      "[400 epo]: TRAIN:[loss=0.002]\n",
      "[450 epo]: TRAIN:[loss=0.002]\n",
      "[500 epo]: TRAIN:[loss=0.002]\n",
      "Finished Training\n",
      "CPU times: user 993 ms, sys: 51.8 ms, total: 1.05 s\n",
      "Wall time: 516 ms\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "label_model.train_model(Ls[0], Y_dev=Ys[1], n_epochs=500, log_train_every=50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can test the quality of our label model on our dev set as a sanity check, but we'll see if we can do better in Step 3 by using the predictions of the label model to train a discriminative model over a larger feature set than just the outputs of these ten labeling functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.879\n",
      "        y=1    y=2   \n",
      " l=1    181    65    \n",
      " l=2    56     698   \n"
     ]
    }
   ],
   "source": [
    "score = label_model.score((Ls[1], Ys[1]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that our trained `LabelModel` outperforms the baseline of taking the majority vote label by approximately 4 accuracy points."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trained Label Model Metrics:\n",
      "Accuracy: 0.879\n",
      "Precision: 0.764\n",
      "Recall: 0.736\n",
      "F1: 0.749\n",
      "        y=1    y=2   \n",
      " l=1    181    65    \n",
      " l=2    56     698   \n"
     ]
    }
   ],
   "source": [
    "print('Trained Label Model Metrics:')\n",
    "scores = label_model.score((Ls[1], Ys[1]), metric=['accuracy','precision', 'recall', 'f1'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Majority Label Voter Metrics:\n",
      "Accuracy: 0.836\n",
      "Precision: 0.623\n",
      "Recall: 0.841\n",
      "F1: 0.716\n",
      "        y=1    y=2   \n",
      " l=1    207    39    \n",
      " l=2    125    629   \n"
     ]
    }
   ],
   "source": [
    "from metal.label_model.baselines import MajorityLabelVoter\n",
    "\n",
    "mv = MajorityLabelVoter(seed=123)\n",
    "print('Majority Label Voter Metrics:')\n",
    "scores = mv.score((Ls[1], Ys[1]), metric=['accuracy','precision', 'recall', 'f1'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that our trained `LabelModel` outperforms the baseline of taking the majority vote label by approximately 0.04 in accuracy and 0.03 in F1 on the dev set. However, it has lower recall. Which of these metrics matters most will vary by application."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Concept: Classifier Base Class"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Both the `LabelModel` and `EndModel` in MeTaL are children of the parent `Classifier` class. The `Classifier` class defines three important methods related to making predictions:\n",
    "* `predict_proba()`: Returns an [n, k+1] numpy array of probabilistic labels (k+1 because of the option to abstain)\n",
    "* `predict()`: Returns an [n]-dim numpy array of  int labels in {0,...,k+1}\n",
    "* `score()`: Returns the score (default: accuracy) of the predictions with respect to target labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Following the data programming paradigm, we will use our label model's probabilistic (float) predictions as training labels for a discriminative end model. Because we want probabilistic labels, we use the `predict_proba()` method here. The predictions are in the form of an [n,k] np.ndarray of floats summing to 1.0 in each row, where `Y_ps[i,j]` is the predicted probability of the $i$th data point having true label $j$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.33879477, 0.66120523],\n",
       "       [0.01750565, 0.98249435],\n",
       "       [0.02757499, 0.97242501],\n",
       "       ...,\n",
       "       [0.74142135, 0.25857865],\n",
       "       [0.98866597, 0.01133403],\n",
       "       [0.38616857, 0.61383143]])"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Y_train_ps stands for \"Y[labels]_train[split]_p[redicted]s[oft]\"\n",
    "Y_train_ps = label_model.predict_proba(Ls[0])\n",
    "Y_train_ps"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analysis tools"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, we may want to analyze our LabelModel's predictions so that we better understand what our model is learning."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can take a look at the confusion matrix to get a sense for class size, error rates, and commonly confused classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "        y=1    y=2   \n",
      " l=1    181    56    \n",
      " l=2    65     698   \n"
     ]
    }
   ],
   "source": [
    "from metal.analysis import confusion_matrix\n",
    "\n",
    "Y_dev_p = label_model.predict(Ls[1])\n",
    "\n",
    "cm = confusion_matrix(Ys[1], Y_dev_p)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If matplotlib is installed, we can also use some of the visualization tools provided in the `contrib/` directory to plot the label distributions.  \n",
    "First, we observe that our problem has a severe class imbalance, which our classifier captures in a general sense.  \n",
    "Second, we can look at how confident our model is in its predictions for class 1 and see that most have probabilities very close to 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAH2NJREFUeJzt3XucVWW9x/HPVwQBb1xEQ+4qIVaKOCrkJRO7qAlakdpFMopTalaWhp5TYmnZTctKjLTEu2YaZGZ6UI6VVxQ0FdFRQUYURgQEAQP6nT/2M7IZFjN7mFmzN8z3/Xrt117rWc9a67fnNcyXZ922IgIzM7P6til3AWZmVpkcEGZmlskBYWZmmRwQZmaWyQFhZmaZHBBmZpbJAWFbNEnTJX2xtddN6x8mac7mrp+xvb9KGpOmPy/pHy247c9IurultmdtgwPCKoKkuZKOKncddSRNkLRG0vL0ek7SryT1rOsTEX+PiEElbuu6xvpFxNERMbkFau8vKSRtW7Tt6yPiw83dtrUtDgizTbs5InYEugEnAO8CHisOiZagAv9btIrjX0qraJK6SrpDUq2kJWm6d71ue0p6RNIySVMkdStaf5ikByQtlfSEpCOaWkNErImIp4ETgVrgm2nbR0iqKdrXtyW9kkYccySNkPRR4DzgREkrJD2R+k6XdJGkfwIrgT0yDnlJ0i/T53pW0oiiBRuMuOqNUu5P70vTPofXP2Ql6f2SHk3bflTS+4uWTZf0fUn/TJ/lbkm7NPXnZls+B4RVum2A3wP9gL7AKuBX9fqcAnwB2B1YC1wGIKkX8BfgQgqjgG8Bf5TUY3MKiYh1wBTgsPrLJA0CzgAOTKOOjwBzI+Iu4AcURiM7RMR+Rat9DhgH7AjMy9jlwcCLwC7A+cBtxeHXgMPTe5e0zwfr1dqNws/lMqA7cAnwF0ndi7p9GjgV2BXoQOFnZ22MA8IqWkQsjog/RsTKiFgOXAR8oF63ayPiqYh4C/gO8ClJ7YDPAndGxJ0R8Z+IuAeYARzTjJIWUAib+tYB2wH7SGofEXMj4oVGtnV1RDwdEWsjYk3G8kXAz9MI5mZgDnBsM2qvcyzwfERcm/Z9I/AscFxRn99HxHMRsQq4BRjSAvu1LYwDwiqapM6SfiNpnqQ3KRw+6ZICoM78oul5QHsK/+vuB4xOh5eWSloKHAo05xxCL+CN+o0RUQ18HZgALJJ0k6TdG9nW/EaWvxIbPk1zHoVRUnPtzsYjlnkUPlud14qmVwI7tMB+bQvjgLBK901gEHBwROzE+sMnKurTp2i6L7AGeJ3CH+BrI6JL0Wv7iLh4cwpJJ5KPA/6etTwiboiIQykEUwA/qlu0iU029ijlXpKKP2dfCiMYgLeAzkXL3tWE7S5INRbrC7zSyHrWxjggrJK0l9Sx6LUthePzqyiccO1G4Vh8fZ+VtI+kzsD3gFvT+YLrgOMkfURSu7TNIzJOcjdIUntJg4EbKfwhviSjzyBJR0raDlidal6XFi8E+m/GlUq7Amem/Y8GBgN3pmWzgJPSsirgk0Xr1QL/AfbYxHbvBN4t6dOStpV0IrAPcEcT67OtnAPCKsmdFP6w1r0mAD8HOlEYETwE3JWx3rXA1RQOi3QEzgSIiPnAKApXEdVSGFGcTem/9ydKWgEsBaYCi4EDImJBRt/tgItTna9R+ON+Xlr2h/S+WNLjJe4b4GFgYNrmRcAnI2JxWvYdYE9gCXABcEPdShGxMvX/Zzq0Nqx4o2kbH6MwOlsMnAN8LCJeb0Jt1gbIXxhkZmZZPIIwM7NMDggzM8uUa0BI+oakpyU9JenGdJJwgKSHJT0v6WZJHVLf7dJ8dVreP8/azMysYbkFRLqL9UygKiLeC7QDTqJw6d+lETGQwgm2sWmVscCSiNgLuJT1lwiamVkZbNt4l2Zvv5OkNRSu2X4VOJLCbfwAkylcqTKRwtUmE1L7rcCvJCkaOIu+yy67RP/+/XMp3Mxsa/XYY4+9HhGNPnImt4CIiFck/RR4mcIli3cDjwFLI2Jt6lbD+rs3e5HuLI2ItZKWUXhOzAaX3kkaR+H5NfTt25cZM2bk9RHMzLZKkrKe/bWRPA8xdaUwKhhA4db+7YGjM7rWjRDUwLL1DRGTIqIqIqp69NisZ66ZmVkJ8jxJfRTwUkTUpgeR3Qa8n8JzdOpGLr1Z/+iAGtIjE9Lyncl45o2ZmbWOPAPiZWBYetiagBHAM8B9rH8swBgKj0+Gwp2qY9L0J4F7Gzr/YGZm+crzHMTDkm4FHqfwjP6ZwCQKz6G/SdKFqe2qtMpVwLWSqimMHE7KqzYzq2xr1qyhpqaG1atXl7uULVrHjh3p3bs37du336z1t+hHbVRVVYVPUpttfV566SV23HFHunfvzoYPtLVSRQSLFy9m+fLlDBgwYINlkh6LiKrGtuE7qc2s4qxevdrh0EyS6N69e7NGYQ4IM6tIDofma+7P0AFhZmaZ8r6T2sys2fqP/0uLbm/uxQ1/tXe7du143/vex9q1axk8eDCTJ0+mc+fODa6zKdOnT+enP/0pd9xxB1OnTuWZZ55h/PjxmX2XLl3KDTfcwGmnndakfUyYMIEddtiBb33rW5tV46Y4IMwqVEv/UdyS/HZkT9bULC3b/jt16sSsWbMA+MxnPsMVV1zBWWed9c7yiCAi2Gabph2EGTlyJCNHjtzk8qVLl3L55Zc3OSDy4kNMZmYNOOyww6iurmbu3LkMHjyY0047jaFDhzJ//nzuvvtuhg8fztChQxk9ejQrVqwA4K677mLvvffm0EMP5bbbbntnW1dffTVnnHEGAAsXLuSEE05gv/32Y7/99uOBBx5g/PjxvPDCCwwZMoSzzz4bgJ/85CcceOCB7Lvvvpx//vpv3L3ooosYNGgQRx11FHPmzMnls3sEYWZtz4KZDS+P/8CCmaxdu5a/3n4THz3i/bDwaebMmcPvfzyey//nS7z+xgtc+N3x/O91v2L7zp340a+v5pILzuacr4zhS1/4PPfe8hv2GtCHE788HlavKuxzyTx4qxYWzOTML3+bDxywL7f/+rusW7eOFW+t5OJvfI6nZs1g1p2/B+DuGy/n+Sce5JE//YaIYOTnv879t/Vh+86duOmmm5g5s1Dj0KFDOeCAA1r8x+SAMDOrZ9XqtxnyocK9uocdvD9jTz6eBQtr6de7J8MO2BeAhx77F8889xKHjDoVgH+vWcPwA/bl2eq5DOi7OwP36AvAZz9xDJOu++NG+7j3n49yzS++DxTOeey8044sWbZ8gz53/99D3P1/D7H/h08GYMXKlTz/0nyWr3iLE0444Z3zIg0dtmoOB4SZWT2dOm7HrHtu2qh9+86d3pmOCD50+MHcePkPN+gz66k5LXaJbkRw7hmn8l+f++QG7T//7fWtchmwz0GYmW2GYQfsyz8ffYLql14GYOWqVTz3wjz23qs/L728gBfmzgfgxj/dlbn+iEMPYuI1fwBg3bp1vLl8BTtu35nlK956p89HjhjO726eyoq3VgLwyquLWPT6Gxw+bCi33347q1atYvny5fz5z3/O5TN6BGFmFW/qGYew7zYvlbuMDfTo3pWrL53Ayaefx9v//jcAF55zOu/esx+TfvzfHHvK19ilWxcOPWgITz1bvdH6v/je2Yw750KuumkK7bbZhok/PJfhVftxyIFDeO+Rozn6g+/nJ9/5BrOff4nhIz8PwA6dO3HdLy9k6PsGc+KJJzJkyBD69evHYYcdlstn9LOYzCpUW7/Mdbe+e2zQVmkBUXa7719St9mzZzN48OAN2vwsJjMzaxYHhJmZZXJAmJlZJgeEmZllckCYmVkmB4SZmWXK7T4ISYOAm4ua9gC+C1yT2vsDc4FPRcQSFW4L/AVwDLAS+HxEPJ5XfWa25dj3yn4tu8Fx0xvtsrB2Md+Y8DMeevxfdN15Rzq0b885p43hhKOPzOw//YEZ/PSKa7jjmss2Wtb/4GOZ8dfr2KVb12YW3rpyG0FExJyIGBIRQ4ADKPzRvx0YD0yLiIHAtDQPcDQwML3GARPzqs3MrCERwfFfOIvDDx7Kiw/+mcfuuoGbJv6QmlcXlbu0VtVad1KPAF6IiHmSRgFHpPbJwHTg28Ao4Joo3Ln3kKQuknpGxKutVKOZGQD3/uMROnRoz5dPWf8MpH69d+erXziJ1avf5ivn/oAZT85m23btuOT8s/jgIQdusP7iN5Zy8unnUbt4CQcNeQ9b6g3JrXUO4iTgxjS9W90f/fS+a2rvBcwvWqcmtW1A0jhJMyTNqK2tzbFkM2urnn7uRYa+d+/MZb+++hYA/jXtFm68/AeM+fr5rF799gZ9Lrh0EoceNISZd9/IyA9/gJdfeS33mvOQ+whCUgdgJHBuY10z2jaK3YiYBEyCwqM2ml2gmVkjTj/vh/zjkVl06NCe3j1346unngjA3nsNoF/vd/Hci/M26H//Q49z25U/BeDYow6ja5edWr3mltAaI4ijgccjYmGaXyipJ0B6rzuoVwP0KVqvN7CgFeozM9vAe969B48/9ew787/+wblMu+UKahcvKflwUWs8jjtvrREQJ7P+8BLAVGBMmh4DTClqP0UFw4BlPv9gZuVw5KEHsfrtfzNx8h/eaVu5ajUAhx88lOtv/ysAz70wj5dfeY1Be/bfYP3Dhw3l+tsKff567z9ZsvTN1im8heV6iElSZ+BDwH8VNV8M3CJpLPAyMDq130nhEtdqClc8nZpnbWa25Xjyi/Na9WmukvjTVT/jGxN+xo8nTqZH965s36kTPzrvTEZ95Ai+PP4HvG/Ep9i2XTuuvvQCttuuwwbrn/+NcZx8+nkM/cin+cCwofTt9a5Wq70l+XHfZhXKj/v2474b5Md9m5lZuTggzMwskwPCzCpOEFvszWWVpLk/QweEmVWceUvXsHblmw6JZogIFi9eTMeOHTd7G631qA0zs5L98uElfBXo1+V1lO6hnS0/OWEDy2Y32qVjx4707t17s3fhgDCzivPm2//hovsXb9A2t+Ony1RNhZqwLPdd+BCTmZllckCYmVkmB4SZmWVyQJiZWSYHhJmZZXJAmJlZJgeEmZllckCYmVkmB4SZmWVyQJiZWSYHhJmZZXJAmJlZplwDQlIXSbdKelbSbEnDJXWTdI+k59N719RXki6TVC3pSUlD86zNzMwalvcI4hfAXRGxN7AfMBsYD0yLiIHAtDQPcDQwML3GARNzrs3MzBqQW0BI2gk4HLgKICL+HRFLgVHA5NRtMnB8mh4FXBMFDwFdJPXMqz4zM2tYniOIPYBa4PeSZkq6UtL2wG4R8SpAet819e8FzC9avya1bUDSOEkzJM2orfUXiJiZ5SXPgNgWGApMjIj9gbdYfzgpizLaNvq+wYiYFBFVEVHVo0ePlqnUzMw2kmdA1AA1EfFwmr+VQmAsrDt0lN4XFfXvU7R+b2BBjvWZmVkDcguIiHgNmC9pUGoaATwDTAXGpLYxwJQ0PRU4JV3NNAxYVncoyszMWl/e30n9VeB6SR2AF4FTKYTSLZLGAi8Do1PfO4FjgGpgZeprZmZlkmtARMQsoCpj0YiMvgGcnmc9ZmZWOt9JbWZmmRwQZmaWyQFhZmaZHBBmZpbJAWFmZpkcEGZmlskBYWZmmRwQZmaWyQFhZmaZHBBmZpbJAWFmZpkcEGZmlskBYWZmmRwQZmaWyQFhZmaZHBBmZpap0YCQtL2kbdL0uyWNlNQ+/9LMzKycShlB3A90lNQLmEbhq0CvzrMoMzMrv1ICQhGxEvg48MuIOAHYp5SNS5or6V+SZkmakdq6SbpH0vPpvWtql6TLJFVLelLS0M39UGZm1nwlBYSk4cBngL+ktqZ8l/UHI2JIRNR9N/V4YFpEDKQwIhmf2o8GBqbXOGBiE/ZhZmYtrJSA+BpwLnB7RDwtaQ/gvmbscxQwOU1PBo4var8mCh4Cukjq2Yz9mJlZMzQ6EoiI+ymch6ibfxE4s8TtB3C3pAB+ExGTgN0i4tW0rVcl7Zr69gLmF61bk9peLd6gpHEURhj07du3xDLMzKypGg0ISe8GvgX0L+4fEUeWsP1DImJBCoF7JD3b0K4y2mKjhkLITAKoqqraaLmZmbWMUs4l/AG4ArgSWNeUjUfEgvS+SNLtwEHAQkk90+ihJ7Aoda8B+hSt3htY0JT9mZlZyynlHMTaiJgYEY9ExGN1r8ZWSvdP7Fg3DXwYeAqYCoxJ3cYAU9L0VOCUdDXTMGBZ3aEoMzNrfaWMIP4s6TTgduDtusaIeKOR9XYDbpdUt58bIuIuSY8Ct0gaC7wMjE797wSOAaqBlRTutzAzszIpJSDq/rd/dlFbAHs0tFI6mb1fRvtiYERGewCnl1CPmZm1glKuYhrQGoWYmVllKeUqpvbAV4DDU9N0CpesrsmxLjMzK7NSDjFNBNoDl6f5z6W2L+ZVlJmZlV8pAXFgRBSfS7hX0hN5FWRmZpWhlMtc10nas24mPWqjSfdDmJnZlqeUEcTZwH2SXqRwt3M/fAmqmdlWr5SrmKZJGggMohAQz0bE242sZmZmW7hNBoSkIyPiXkkfr7doT0lExG0512ZmZmXU0AjiA8C9wHEZywJwQJiZbcU2GRARcX6a/F5EvFS8TJJvnjMz28qVchXTHzPabm3pQszMrLI0dA5ib+A9wM71zkPsBHTMuzAzMyuvhs5BDAI+BnRhw/MQy4Ev5VmUmZmVX0PnIKYAUyQNj4gHW7EmMzOrAKWcg/iypC51M5K6SvpdjjWZmVkFKCUg9o2IpXUzEbEE2D+/kszMrBKUEhDbSOpaNyOpG6U9osPMzLZgpfyh/xnwgKS6S1tHAxflV5KZmVWCRkcQEXEN8AlgIbAI+HhEXFvqDiS1kzRT0h1pfoCkhyU9L+lmSR1S+3Zpvjot7785H8jMzFrGJgNC0k7pvRvwGnADcD3wWmor1deA2UXzPwIujYiBwBJgbGofCyyJiL2AS1M/MzMrk4ZGEDek98eAGUWvuvlGSeoNHAtcmeYFHMn6O7EnA8en6VFpnrR8ROpvZmZl0NB9EB9L78157tLPgXOAHdN8d2BpRKxN8zVArzTdC5if9rlW0rLU//XiDUoaB4wD6Nu3bzNKMzOzhjT0qI2hDa0YEY83tFzSx4BFEfGYpCPqmrM2VcKy4v1OAiYBVFVVbbTczMxaRkNXMf0svXcEqoAnKPwR3xd4GDi0kW0fAoyUdEzaxk4URhRdJG2bRhG9gQWpfw3QB6iRtC2wM/BGkz+RmZm1iE2eg4iID0bEB4F5wNCIqIqIAyjcJFfd2IYj4tyI6B0R/YGTgHsj4jPAfcAnU7cxwJQ0PTXNk5bfGxEeIZiZlUkpN8rtHRH/qpuJiKeAIc3Y57eBsyRVUzjHcFVqvwrontrPAsY3Yx9mZtZMpdwoN1vSlcB1FM4JfJYNL1ttVERMB6an6ReBgzL6rKZwE56ZmVWAUgLiVOArFO5nALgfmJhbRWZmVhEaDYiIWC3pCuDOiJjTCjWZmVkFaPQchKSRwCzgrjQ/RNLUvAszM7PyKuUk9fkUzhksBYiIWUD/HGsyM7MKUEpArI2IZblXYmZmFaWUk9RPSfo00E7SQOBM4IF8yzIzs3IrZQTxVeA9wNsUHuC3DPh6nkWZmVn5NTiCkNQOuCAizgb+u3VKMjOzStDgCCIi1gEHtFItZmZWQUo5BzEzXdb6B+CtusaIuC23qszMrOxKCYhuwGIKX/RTJwAHhJnZVqyUgDg7Il5vvJuZmW1NGvpO6uMk1QJPSqqR9P5WrMvMzMqsoZPUFwGHRcTuwCeAH7ZOSWZmVgkaCoi1EfEsQEQ8zPrvlTYzszagoXMQu0o6a1PzEXFJfmWZmVm5NRQQv2XDUUP9eTMz24ptMiAi4oLWLMTMzCpLKc9i2iySOkp6RNITkp6WdEFqHyDpYUnPS7pZUofUvl2ar07L++dVm5mZNS63gKDwcL8jI2I/YAjwUUnDgB8Bl0bEQGAJMDb1HwssiYi9gEtTPzMzK5PcAiIKVqTZ9ukVFO7IvjW1TwaOT9Oj0jxp+QhJyqs+MzNrWClfOfo/RdPbNWXjktpJmgUsAu4BXgCWRsTa1KUG6JWmewHzAdLyZUD3jG2OkzRD0oza2tqmlGNmZk3Q0J3U50gaDnyyqPnBpmw8ItZFxBCgN4WvLR2c1a1ulw0sK97mpIioioiqHj16NKUcMzNrgoZGEHOA0cAekv4uaRLQXdKgpu4kIpYC04FhQBdJdVdP9QYWpOkaoA9AWr4z8EZT92VmZi2joYBYApwHVANHAJel9vGSGv3KUUk9JHVJ052Ao4DZwH2sH5WMAaak6alpnrT83ojYaARhZmato6Eb5T4KnA/sCVwCPAG8FRGnlrjtnsDk9K102wC3RMQdkp4BbpJ0ITATuCr1vwq4VlI1hZHDSU3+NGZm1mIaulHuPABJTwDXAfsDPST9g8LlqMc1tOGIeDKtU7/9RQrnI+q3r6ZwSMvMzCpAKd8H8beIeBR4VNJXIuJQSbvkXZiZmZVXo5e5RsQ5RbOfT23+AiEzs61ck26Ui4gn8irEzMwqS56P2jAzsy2YA8LMzDI5IMzMLJMDwszMMjkgzMwskwPCzMwylXKjnLUFE3YudwWVZcKycldgVnYeQZiZWaY2O4LoP/4v5S6hosztWO4KzKzSeARhZmaZHBBmZpbJAWFmZpkcEGZmlskBYWZmmRwQZmaWKbeAkNRH0n2SZkt6WtLXUns3SfdIej69d03tknSZpGpJT0oamldtZmbWuDxHEGuBb0bEYGAYcLqkfYDxwLSIGAhMS/MARwMD02scMDHH2szMrBG5BUREvBoRj6fp5cBsoBcwCpicuk0Gjk/To4BrouAhoIuknnnVZ2ZmDWuVcxCS+gP7Aw8Du0XEq1AIEWDX1K0XML9otZrUZmZmZZB7QEjaAfgj8PWIeLOhrhltkbG9cZJmSJpRW1vbUmWamVk9uQaEpPYUwuH6iLgtNS+sO3SU3hel9hqgT9HqvYEF9bcZEZMioioiqnr06JFf8WZmbVyeVzEJuAqYHRGXFC2aCoxJ02OAKUXtp6SrmYYBy+oORZmZWevL82muhwCfA/4laVZqOw+4GLhF0ljgZWB0WnYncAxQDawETs2xNjMza0RuARER/yD7vALAiIz+AZyeVz1mZtY0vpPazMwyOSDMzCyTA8LMzDI5IMzMLJMDwszMMjkgzMwskwPCzMwyOSDMzCyTA8LMzDI5IMzMLJMDwszMMjkgzMwskwPCzMwyOSDMzCyTA8LMzDI5IMzMLJMDwszMMjkgzMwsU24BIel3khZJeqqorZukeyQ9n967pnZJukxStaQnJQ3Nqy4zMytNniOIq4GP1msbD0yLiIHAtDQPcDQwML3GARNzrMvMzEqQW0BExP3AG/WaRwGT0/Rk4Pii9mui4CGgi6SeedVmZmaNa+1zELtFxKsA6X3X1N4LmF/Urya1mZlZmVTKSWpltEVmR2mcpBmSZtTW1uZclplZ29XaAbGw7tBRel+U2muAPkX9egMLsjYQEZMioioiqnr06JFrsWZmbVlrB8RUYEyaHgNMKWo/JV3NNAxYVncoyszMymPbvDYs6UbgCGAXSTXA+cDFwC2SxgIvA6NT9zuBY4BqYCVwal51mZlZaXILiIg4eROLRmT0DeD0vGoxM7Omq5ST1GZmVmEcEGZmlskBYWZmmRwQZmaWyQFhZmaZHBBmZpbJAWFmZpkcEGZmlskBYWZmmRwQZmaWyQFhZmaZHBBmZpbJAWFmZpkcEGZmlskBYWZmmRwQZmaWyQFhZmaZHBBmZpbJAWFmZpkqKiAkfVTSHEnVksaXux4zs7asYgJCUjvg18DRwD7AyZL2KW9VZmZtV8UEBHAQUB0RL0bEv4GbgFFlrsnMrM3attwFFOkFzC+arwEOrt9J0jhgXJpdIWlOK9S21RPsArxe7joqxgUqdwVWj39H62ne72i/UjpVUkBkfdrYqCFiEjAp/3LaFkkzIqKq3HWYbYp/R1tfJR1iqgH6FM33BhaUqRYzszavkgLiUWCgpAGSOgAnAVPLXJOZWZtVMYeYImKtpDOAvwHtgN9FxNNlLqst8WE7q3T+HW1litjoML+ZmVlFHWIyM7MK4oAwM7NMDog2TtLvJC2S9FS5azHLIqmPpPskzZb0tKSvlbumtsLnINo4SYcDK4BrIuK95a7HrD5JPYGeEfG4pB2Bx4DjI+KZMpe21fMIoo2LiPuBN8pdh9mmRMSrEfF4ml4OzKbw5AXLmQPCzLYYkvoD+wMPl7eStsEBYWZbBEk7AH8Evh4Rb5a7nrbAAWFmFU9SewrhcH1E3FbuetoKB4SZVTRJAq4CZkfEJeWupy1xQLRxkm4EHgQGSaqRNLbcNZnVcwjwOeBISbPS65hyF9UW+DJXMzPL5BGEmZllckCYmVkmB4SZmWVyQJiZWSYHhJmZZXJAmJVA0oom9J0g6Vt5bd+stTggzMwskwPCbDNJOk7Sw5JmSvpfSbsVLd5P0r2Snpf0paJ1zpb0qKQnJV1QhrLNSuaAMNt8/wCGRcT+wE3AOUXL9gWOBYYD35W0u6QPAwOBg4AhwAHp+zjMKtK25S7AbAvWG7g5faFNB+ClomVTImIVsErSfRRC4VDgw8DM1GcHCoFxf+uVbFY6B4TZ5vslcElETJV0BDChaFn9Z9gEIOCHEfGb1inPrHl8iMls8+0MvJKmx9RbNkpSR0ndgSOAR4G/AV9I32uApF6Sdm2tYs2ayiMIs9J0llRTNH8JhRHDHyS9AjwEDCha/gjwF6Av8P2IWAAskDQYeLDwBGtWAJ8FFuVfvlnT+WmuZmaWyYeYzMwskwPCzMwyOSDMzCyTA8LMzDI5IMzMLJMDwszMMjkgzMws0/8DZOE0CX06nUcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAHpRJREFUeJzt3XmYXFWd//H3h4AgsgQIaAjRiAYRGIkYEMYNwVHAJeiAgiiLaERl1FFRcHnUQRScQZRRwQgILiiLC1FxlB+LiAqaQFgjPyICCYkgW1gCDAmf+eOe1qK53V2d9K2qdH9ez9NP33vuube+ddKpb51z7iLbRERE9LdGtwOIiIjelAQRERG1kiAiIqJWEkRERNRKgoiIiFpJEBERUSsJIlaJpM9I+u5K7nuwpMsG2X6JpHeW5QMk/WolX+d6SbuuzL4r+XqW9NxO71v2/7ikU1Z2/5rjPShpy7J8uqTPjeCxT5b0qZE6Xoy8JIgxSNItkh4u//nvkPQtSet1O67B2P6e7VcPVa/uQ8z2trYvGc7rSZpSPqzXHGaojSkJ8xFJD0i6X9JcSUdKWruvju3P235nm8casp7t9WzfPAKxP+nLgO3DbB+9qseO5iRBjF2vt70esAOwI/DJ/hVUyd9Ibznc9vrARODDwH7A+ZI0ki/SS4kxuif/+cc427cDvwC2g79/szxG0m+BZcCWkjaXNFvSPZIWSHpXv8OsI+ms8s32Sknb920o33D/XLbdIOmN/faVpP+WtFTSnyTtXhdn6zfQkrhOkHRn2e8aSdtJmgkcAHy09I5+WurfIulVZXlcGYbpi2mupMnDaTNJO0n6vaT7JC2R9FVJT+lXbS9JN0u6S9J/tiZaSe+QNF/SvZJ+KelZw3l9ANsPlV7RG4BdgNeWY/99yE/SOpK+K+nuEusfJT1d0jHAy4Cvlnb6aqlvSe+TdBNwU0tZ65DXBEkXlLb7dV/sdT2uvl6KpOcDJwO7lNe7r2x/Qm9P0rvK39c95e9t85ZtlnSYpJtKu31tpJNiPFkSxBhXPhz3Aq5qKX47MBNYH7gV+D6wCNgc2Af4fL8P8hnAOcDGwJnATyStVbb9merDaEPgs8B3JU1s2ffFwM3ABODTwI8kbTxE2K8GXg5sBYwH3gLcbXsW8D3gi2Vo5PU1+34I2L+85w2Ad1AlwuFYAfx7iXkXYHfgvf3qvBGYTtVDm1FeB0l7Ax8H3gRsCvyGqn1Xiu3bgDlUbdzfQVTtPhnYBDgMeNj2J8rrHl7a6fCWffam+jfZZoCXPAA4muq9z6Nq76FinF9e+/fl9cb3ryNpN+ALwJupeke3Aj/oV+11VL3d7Uu91wz12rFqkiDGrp+Ub3KXAb8GPt+y7XTb19teDjwDeCnwMduP2J4HnEKVRPrMtX2u7ceALwHrADsD2D7H9mLbj9s+i+qb6U4t+94JfNn2Y2X7jZRvw4N4jCp5bQ3I9nzbS9p83+8EPmn7Rleutn13m/tS3tNc25fbXm77FuAbwCv6VTvO9j3lA/zLVEkJ4N3AF0rMy6nafdrK9CJaLKZKzv09RpUYnmt7RYn7/iGO9YUS98MDbP+57UttPwp8gqpXMKwe2AAOAE6zfWU59lHl2FNa6hxr+77SphcD00bgdWMQSRBj1962x9t+lu339vtAWNiyvDlwj+0HWspuBSbV1bf9OP/obSDpQEnzyhDHfVRDWRNa9r3dT7xj5K19+w7E9kXAV4GvAXdImiVpg6HecDGZqlez0iRtJelnkv4q6X6qD/kJ/aq1tmHre3oW8JWW9rgHEE9sz+GaVI7T33eAXwI/kLRY0hdbenYDWdjudtsPltcd9N+rTZtTtVPrse/mie3y15blZUBPn1gxGiRBRJ3WD+zFwMaS1m8peyZwe8v6379BlrH2LYDF5VvxN4HDgU3K0MJ1VB+IfSb1G0t+ZnnNwQO0T7T9ImBbqqGmI2pir7MQeM5Qxx/CScCfgKm2N6AaMuo/Ht76rbr1PS0E3l2Sc9/PU23/bmUCKd/eX0Q1ZPQEpVf2WdvbAP9MNURzYN/mAQ45VPu1/luvR9VzWQw8VIrXban7jGEcdzFV8uw79tOoej+3D7hHNC4JIgZleyHwO+ALZdLzBcChPHHs+UWS3lQmKD8IPApcDjyN6oPhbwCSDqFMhrfYDHi/pLUk7Qs8Hzh/sJgk7SjpxeXb8EPAI1TzAgB3AFsOsvspwNGSppbJ7hdI2mSQ+muX9933swbV8Nb9wIOStgbeU7PfEZI2Kh/gHwDOKuUnA0dJ2ra8lw3L+x4WSetKegVwHvAHatpM0isl/ZOkcSXex2i/nQayl6SXlkn5o4ErbC+0/TeqD/O3qToR4B08MRHfAWxRM5nf50zgEEnTVJ22+/ly7FtWIsYYIUkQ0Y79gSlU3/J+DHza9gUt28+jmii+l2pu4k3l2+sNwPHA76k+IP4J+G2/Y18BTAXuAo4B9mljTmADqp7JvVTDEncD/1W2nQpsU4ZwflKz75eAs4FfUX1ongo8dZDXehB4uOVnN+AjwFuBB0ocZ9Xsdx4wl2oi9+fldbD9Y+A4qmGf+6l6VHsO8X5bfVXSA1Tt+WXgh8AeZWivv2cA55b3OZ9qrqnvosavAPuUM4JOHMbrn0l1MsE9VD2XA1q2vYuqJ3c3Vc+utVd0EXA98FdJd/U/qO0LgU+V97OEKrnsN4y4ogHKA4MiIqJOehAREVErCSIiImolQURERK0kiIiIqLVa35BrwoQJnjJlSrfDiIhYrcydO/cu25sOVW+1ThBTpkxhzpw53Q4jImK1IunWoWtliCkiIgaQBBEREbWSICIiolYSRERE1EqCiIiIWkkQERFRKwkiIiJqJUFEREStJIiIiKi1Wl9Jfe3tS5ly5M9X6Ri3HPvaEYomImJ0SQ8iIiJqNZYgyvN7/yDpaknXS/psKT9d0l8kzSs/00q5JJ0oaYGkayTt0FRsERExtCaHmB4FdrP9YHm4/GWSflG2HWH73H7196R6NvFU4MXASeV3RER0QWM9CFceLKtrlZ/BHoA9A/h22e9yYLykiU3FFxERg2t0DkLSOEnzgDuBC2xfUTYdU4aRTpC0dimbBCxs2X1RKet/zJmS5kias2LZ0ibDj4gY0xpNELZX2J4GbAHsJGk74Chga2BHYGPgY6W66g5Rc8xZtqfbnj5u3Q0bijwiIjpyFpPt+4BLgD1sLynDSI8C3wJ2KtUWAZNbdtsCWNyJ+CIi4smaPItpU0njy/JTgVcBf+qbV5AkYG/gurLLbODAcjbTzsBS20uaii8iIgbX5FlME4EzJI2jSkRn2/6ZpIskbUo1pDQPOKzUPx/YC1gALAMOaTC2iIgYQmMJwvY1wAtryncboL6B9zUVT0REDE+upI6IiFpJEBERUSsJIiIiaiVBRERErSSIiIiolQQRERG1kiAiIqJWEkRERNRKgoiIiFpJEBERUSsJIiIiaiVBRERErSSIiIiolQQRERG1kiAiIqJWEkRERNRKgoiIiFpJEBERUSsJIiIiaiVBRERErcYShKR1JP1B0tWSrpf02VL+bElXSLpJ0lmSnlLK1y7rC8r2KU3FFhERQ2uyB/EosJvt7YFpwB6SdgaOA06wPRW4Fzi01D8UuNf2c4ETSr2IiOiSxhKEKw+W1bXKj4HdgHNL+RnA3mV5RlmnbN9dkpqKLyIiBtfoHISkcZLmAXcCFwB/Bu6zvbxUWQRMKsuTgIUAZftSYJOaY86UNEfSnBXLljYZfkTEmNZogrC9wvY0YAtgJ+D5ddXK77regp9UYM+yPd329HHrbjhywUZExBN05Cwm2/cBlwA7A+MlrVk2bQEsLsuLgMkAZfuGwD2diC8iIp6sybOYNpU0viw/FXgVMB+4GNinVDsIOK8szy7rlO0X2X5SDyIiIjpjzaGrrLSJwBmSxlElorNt/0zSDcAPJH0OuAo4tdQ/FfiOpAVUPYf9GowtIiKG0FiCsH0N8MKa8pup5iP6lz8C7NtUPBERMTy5kjoiImolQURERK0kiIiIqJUEERERtZIgIiKiVhJERETUSoKIiIhaSRAREVErCSIiImolQURERK0kiIiIqJUEERERtZIgIiKiVhJERETUSoKIiIhaSRAREVErCSIiImolQURERK0kiIiIqJUEERERtRpLEJImS7pY0nxJ10v6QCn/jKTbJc0rP3u17HOUpAWSbpT0mqZii4iIoa3Z4LGXAx+2faWk9YG5ki4o206w/V+tlSVtA+wHbAtsDvw/SVvZXtFgjBERMYDGehC2l9i+siw/AMwHJg2yywzgB7Yftf0XYAGwU1PxRUTE4DoyByFpCvBC4IpSdLikaySdJmmjUjYJWNiy2yJqEoqkmZLmSJqzYtnSBqOOiBjbGk8QktYDfgh80Pb9wEnAc4BpwBLg+L6qNbv7SQX2LNvTbU8ft+6GDUUdERGNJghJa1Elh+/Z/hGA7Ttsr7D9OPBN/jGMtAiY3LL7FsDiJuOLiIiBDZkgJD1N0hpleStJbygf/EPtJ+BUYL7tL7WUT2yp9kbgurI8G9hP0tqSng1MBf7Q/luJiIiR1M5ZTJcCLytzBRcCc4C3AAcMsd9LgLcD10qaV8o+DuwvaRrV8NEtwLsBbF8v6WzgBqozoN6XM5giIrqnnQQh28skHQr8t+0vSrpqqJ1sX0b9vML5g+xzDHBMGzFFRETD2pmDkKRdqHoMPy9lTV4/ERERPaCdBPEB4Cjgx2UYaEvg4mbDioiIbhuyJ2D7Uqp5iL71m4H3NxlURER035AJQtJWwEeAKa31be/WXFgREdFt7cwlnAOcDJwC5KyiiIgxop0Esdz2SY1HEhERPaWdSeqfSnqvpImSNu77aTyyiIjoqnZ6EAeV30e0lBnYcuTDiYiIXtHOWUzP7kQgERHRW9o5i2kt4D3Ay0vRJcA3bD/WYFwREdFl7QwxnQSsBXy9rL+9lL2zqaAiIqL72kkQO9revmX9IklXNxVQRET0hnbOYloh6Tl9K+VWG7keIiJilGunB3EEcLGkm6nuzvos4JBGo4qIiK5r5yymCyVNBZ5HlSD+ZPvRxiOLiIiuGjBBSNrN9kWS3tRv03Mk0fcI0YiIGJ0G60G8ArgIeH3NNgNJEBERo9iACcL2p8vif9j+S+u28szoiIgYxdo5i+mHNWXnjnQgERHRWwabg9ga2BbYsN88xAbAOk0HFhER3TVYD+J5wOuA8VTzEH0/OwDvGurAkiZLuljSfEnXS/pAKd9Y0gWSbiq/NyrlknSipAWSrpG0w6q+uYiIWHmDzUGcB5wnaRfbv1+JYy8HPmz7SknrA3MlXQAcDFxo+1hJRwJHAh8D9gSmlp8XU93O48Ur8boRETEC2pmDOEzS+L4VSRtJOm2onWwvsX1lWX4AmA9MAmYAZ5RqZwB7l+UZwLdduRwYL2li+28lIiJGUjsJ4gW27+tbsX0v8MLhvIikKWWfK4Cn215SjrUE2KxUmwQsbNltUSnrf6yZkuZImrNi2dLhhBEREcPQToJYo2+eAKo5BNq7RUdf/fWozoT6oO37B6taU+YnFdizbE+3PX3cuhu2G0ZERAxTOx/0xwO/k9R3auu+wDHtHLw8S+KHwPdarry+Q9JE20vKENKdpXwRMLll9y2Axe28TkREjLwhexC2vw38K3AH1Yf5m2x/Z6j9JAk4FZhv+0stm2bzj8eYHgSc11J+YDmbaWdgad9QVEREdN5g10FsYPv+MqT0V+DMlm0b275niGO/hOrhQtdKmlfKPg4cC5wt6VDgNqoeCcD5wF7AAmAZuWNsRERXDTbEdCbVdRBzeeJcgMr6loMd2PZl1M8rAOxeU9/A+wY7ZkREdM5g10G8rvzOfZciIsagwYaYBr2Sue8ah4iIGJ0GG2I6vvxeB5gOXE01ZPQCqusZXtpsaBER0U0DnsVk+5W2XwncCuxQrj14EdUFbws6FWBERHRHOxfKbW372r4V29cB05oLKSIiekE7F8rNl3QK8F2qs5feRnVfpYiIGMXaSRCHAO8BPlDWL6W602pERIxiQyYI249IOhk43/aNHYgpIiJ6wJBzEJLeAMwD/qesT5M0u+nAIiKiu9qZpP40sBNwH4DtecCUBmOKiIge0E6CWG47D16IiBhj2pmkvk7SW4FxkqYC7wd+12xYERHRbe30IP4N2BZ4lOoGfkuBDzYZVEREdN+gPQhJ44DP2j4C+ERnQoqIiF4waA/C9grgRR2KJSIiekg7cxBXldNazwEe6itseYRoRESMQu0kiI2Bu4HdWsoMJEFERIxi7SSII2zf1XgkERHRUwacg5D0ekl/A66RtEjSP3cwroiI6LLBJqmPAV5me3PgX4EvDOfAkk6TdKek61rKPiPpdknzys9eLduOkrRA0o2SXjPcNxIRESNrsASx3PafAGxfAaw/zGOfDuxRU36C7Wnl53wASdsA+1Fdb7EH8PVyim1ERHTJYHMQm0n60EDrtr802IFtXyppSptxzAB+YPtR4C+SFlDd/+n3be4fEREjbLAexDepeg19P/3XV9bhkq4pQ1AblbJJwMKWOotKWUREdMmAPQjbn23g9U4CjqY6TfZo4HjgHYDqQqg7gKSZwEyAcRts2kCIEREB7d2LacTYvsP2CtuPU/VIdiqbFgGTW6puASwe4BizbE+3PX3cuhs2G3BExBjW0QQhaWLL6huBvjOcZgP7SVpb0rOBqcAfOhlbREQ8UTsXyq0USd8HdgUmSFpE9eChXSVNoxo+ugV4N4Dt6yWdDdwALAfeV+4DFRERXTJkgpD0SdufK8trlzONhmR7/5riUwepfwzVtRcREdEDBruS+qOSdgH2aSnOaacREWPEYD2IG4F9gS0l/QaYD2wi6Xm2b+xIdBER0TWDTVLfC3wcWEA1l3BiKT9SUh45GhExyg3Wg9iDamL5OcCXgKuBh2wf0onAIiKiuwbsQdj+uO3dqc42+i5VMtlU0mWSftqh+CIiokvaOc31l7b/CPxR0ntsv1TShKYDi4iI7hryQjnbH21ZPbiU5QFCERGj3LCupLZ9dVOBREREb+norTYiImL1kQQRERG1kiAiIqJWYzfrW11MOfLnq7T/Lce+doQiiYjoLelBRERErSSIiIiolQQRERG1kiAiIqJWEkRERNRKgoiIiFpJEBERUSsJIiIiajWWICSdJulOSde1lG0s6QJJN5XfG5VySTpR0gJJ10jaoam4IiKiPU32IE6neipdqyOBC21PBS4s6wB7AlPLz0zgpAbjioiINjSWIGxfCtzTr3gGcEZZPgPYu6X8265cDoyXNLGp2CIiYmidnoN4uu0lAOX3ZqV8ErCwpd6iUvYkkmZKmiNpzoplSxsNNiJiLOuVSWrVlLmuou1Ztqfbnj5u3Q0bDisiYuzqdIK4o2/oqPy+s5QvAia31NsCWNzh2CIiokWnE8Rs4KCyfBBwXkv5geVspp2BpX1DURER0R2NPQ9C0veBXYEJkhYBnwaOBc6WdChwG7BvqX4+sBewAFgGHNJUXBER0Z7GEoTt/QfYtHtNXQPvayqWiIgYvl6ZpI6IiB6TBBEREbXG/DOpV1WeaR0Ro1V6EBERUSsJIiIiaiVBRERErSSIiIiolQQRERG1kiAiIqJWEkRERNRKgoiIiFpJEBERUSsJIiIiaiVBRERErSSIiIiolQQRERG1kiAiIqJWEkRERNRKgoiIiFpJEBERUasrT5STdAvwALACWG57uqSNgbOAKcAtwJtt39uN+CIioruPHH2l7bta1o8ELrR9rKQjy/rHuhNa56zqI0shjy2NiGb00jOpZwC7luUzgEsYAwkiIqJVLz3nvltzEAZ+JWmupJml7Om2lwCU35vV7ShppqQ5kuasWLa0Q+FGRIw93epBvMT2YkmbARdI+lO7O9qeBcwCWHviVDcVYETEWNeVHoTtxeX3ncCPgZ2AOyRNBCi/7+xGbBERUel4gpD0NEnr9y0DrwauA2YDB5VqBwHndTq2iIj4h24MMT0d+LGkvtc/0/b/SPojcLakQ4HbgH27EFtERBQdTxC2bwa2rym/G9i90/FERES9XEkdERG1kiAiIqJWEkRERNTqpSupIyJWayNx65xekgQxCvTSpfkRMXokQUTEiBgNX1RGWw9gVWUOIiIiaiVBRERErQwxxagwGoY3InpNehAREVErPYjo+sRcvr0H5OmKvSgJIoIMUUXUSYKIrut2DyZGj/wtjazMQURERK30ICJGQIaoYjRKgojoAb0wNJIkFf1liCkiImolQURERK0kiIiIqJUEERERtXpuklrSHsBXgHHAKbaP7XJIEWNCL0yUR2/pqR6EpHHA14A9gW2A/SVt092oIiLGpp5KEMBOwALbN9v+X+AHwIwuxxQRMSb12hDTJGBhy/oi4MWtFSTNBGaW1UdvPe5113Uotl42Abir20F0WdogbdBnTLeDjgOGboNntXOsXksQqinzE1bsWcAsAElzbE/vRGC9LO2QNoC0QZ+0w8i1Qa8NMS0CJresbwEs7lIsERFjWq8liD8CUyU9W9JTgP2A2V2OKSJiTOqpISbbyyUdDvyS6jTX02xfP8guszoTWc9LO6QNIG3QJ+0wQm0g20PXioiIMafXhpgiIqJHJEFERESt1SJBSNpD0o2SFkg6smb72pLOKtuvkDSl81E2q402+JCkGyRdI+lCSW2d57y6GaodWurtI8mSRt3pju20gaQ3l7+H6yWd2ekYm9bG/4dnSrpY0lXl/8Re3YizSZJOk3SnpNprwVQ5sbTRNZJ2GPaL2O7pH6rJ6j8DWwJPAa4GtulX573AyWV5P+CsbsfdhTZ4JbBuWX7PaGuDdtuh1FsfuBS4HJje7bi78LcwFbgK2Kisb9btuLvQBrOA95TlbYBbuh13A+3wcmAH4LoBtu8F/ILq+rKdgSuG+xqrQw+indtvzADOKMvnArtLqrvobnU1ZBvYvtj2srJ6OdU1JKNNu7diORr4IvBIJ4PrkHba4F3A12zfC2D7zg7H2LR22sDABmV5Q0bh9VS2LwXuGaTKDODbrlwOjJc0cTivsTokiLrbb0waqI7t5cBSYJOORNcZ7bRBq0OpvjmMNkO2g6QXApNt/6yTgXVQO38LWwFbSfqtpMvLHZJHk3ba4DPA2yQtAs4H/q0zofWU4X5uPElPXQcxgCFvv9FmndVZ2+9P0tuA6cArGo2oOwZtB0lrACcAB3cqoC5o529hTaphpl2pepK/kbSd7fsajq1T2mmD/YHTbR8vaRfgO6UNHm8+vJ6xyp+Lq0MPop3bb/y9jqQ1qbqUg3W9Vjdt3YJE0quATwBvsP1oh2LrpKHaYX1gO+ASSbdQjbvOHmUT1e3+fzjP9mO2/wLcSJUwRot22uBQ4GwA278H1qG6gd1Yssq3LlodEkQ7t9+YDRxUlvcBLnKZpRklhmyDMrTyDarkMNrGnPsM2g62l9qeYHuK7SlUczFvsD2nO+E2op3/Dz+hOmkBSROohpxu7miUzWqnDW4DdgeQ9HyqBPG3jkbZfbOBA8vZTDsDS20vGc4Ben6IyQPcfkPSfwBzbM8GTqXqQi6g6jns172IR16bbfCfwHrAOWV+/jbbb+ha0A1osx1GtTbb4JfAqyXdAKwAjrB9d/eiHllttsGHgW9K+neqYZWDR9mXRiR9n2oYcUKZa/k0sBaA7ZOp5l72AhYAy4BDhv0ao6zNIiJihKwOQ0wREdEFSRAREVErCSIiImolQURERK0kiIiIqJUEEWOOpBWS5km6TtI5ktYd5v4PDrP+6ZL2qSmfLunEsnywpK+W5cMkHdhSvvlwXi9ipCRBxFj0sO1ptrcD/hc4rHVjubCo8f8btufYfn9N+cm2v11WDwaSIKIrkiBirPsN8FxJUyTNl/R14EpgsqT9JV1behrHte4k6XhJV5Znb2xayt4l6Y+Srpb0w349k1dJ+o2k/y/pdaX+rpKedFNBSZ+R9JHS65gOfK/0eF4r6cct9f5F0o9GvkkiKkkQMWaV+3btCVxbip5HdXvkFwKPAccBuwHTgB0l7V3qPQ240vYOwK+prmAF+JHtHW1vD8ynuh9QnylUN1B8LXCypHWGis/2ucAc4ADb06iujH1+X0KiujL2W8N+4xFtSoKIseipkuZRffjeRnWrFoBby33zAXYELrH9t3IL+e9RPaAF4HHgrLL8XeClZXm70ku4FjgA2LblNc+2/bjtm6jui7T1cIMut4r4DtVtrMcDuzA6b+sePaLn78UU0YCHyzfyvyv3r3qotWgYx+u7X83pwN62r5Z0MNV9cvrXGWi9Xd8Cfkr1MKRzSvKKaER6EBH1rgBeIWmCpHFUzxf4ddm2BtVdgwHeClxWltcHlkhai6oH0WpfSWtIeg7VozJvbDOOB8pxAbC9mOqWzZ+kSkgRjUkPIqKG7SWSjgIupupNnG/7vLL5IWBbSXOpnl74llL+KarEcivVvMb6LYe8kSrBPB04zPYjbT4V93SqOYuHgV1sP0w13LWp7RtW4S1GDCl3c41YzZTrJa6yfeqQlSNWQRJExGqk9FoeAv5llD41MHpIEkRERNTKJHVERNRKgoiIiFpJEBERUSsJIiIiaiVBRERErf8D8+iRWAcd0gQAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "try:\n",
    "    from metal.contrib.visualization.analysis import (\n",
    "        plot_predictions_histogram, \n",
    "        plot_probabilities_histogram,\n",
    "    )\n",
    "\n",
    "    plot_predictions_histogram(Y_dev_p, Ys[1], title=\"Label Distribution\")\n",
    "\n",
    "    Y_dev_ps = label_model.predict_proba(Ls[1])\n",
    "    plot_probabilities_histogram(Y_dev_ps[:,0], title=\"Probablistic Label Distribution\")\n",
    "\n",
    "except ModuleNotFoundError:\n",
    "    print(\"The tools in contrib/visualization/ require matplotlib. Try `conda/pip install matplotlib`.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Train End Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will now construct and train our `EndModel` on the predictions from the LabelModel. Here we describe the construction of a single-task end model. For details on the construction of multi-task end models, refer to the multi-task tutorial.\n",
    "\n",
    "The `EndModel` consists of three components: an *input layer*, zero or more *middle layers*, and a *head layer*, with each layer consisting of a torch.nn.Module followed by various optional additional operators (e.g., a ReLU nonlinearity, batch normalization, and/or dropout). \n",
    "\n",
    "* **Input layer**:\n",
    "The input module is an IdentityModule by default, which simply accepts arbitrary-length feature vectors from X as torch.Tensors and passes them on to the next module in the network. If, however, you would like provide your data in some other format, you may pass in a custom nn.Module which maps your input type to torch.Tensors (for example, if your data points are images, you may pass in a ResNet as your input module).\n",
    "\n",
    "* **Middle layers**\n",
    "The middle layers are layers between the input layer and head layer. Middle modules are nn.Linear by default.\n",
    "\n",
    "* **Head layer**:\n",
    "The head layer is the final layer, consisting of an nn.Linear module (plus a softmax operation when making predictions).\n",
    " \n",
    "When initiated, the EndModel requires a list of output dimensions. \n",
    "The first element is the output dimension of the input module (which is equal to the dimensionality of your feature vectors if using the default Identitymodule).\n",
    "The last element is the cardinality of the classifier.\n",
    "The elements in the middle define the number and dimensionality of any middle layers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "Network architecture:\n",
      "Sequential(\n",
      "  (0): IdentityModule()\n",
      "  (1): Sequential(\n",
      "    (0): Linear(in_features=1000, out_features=10, bias=True)\n",
      "    (1): ReLU()\n",
      "  )\n",
      "  (2): Linear(in_features=10, out_features=2, bias=True)\n",
      ")\n",
      "\n"
     ]
    }
   ],
   "source": [
    "from metal.end_model import EndModel\n",
    "import torch\n",
    "if torch.cuda.is_available():\n",
    "    device = 'cuda'\n",
    "else:\n",
    "    device='cpu'\n",
    "end_model = EndModel([1000,10,2], seed=123, device=device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once initiated, the network structure is printed so you can confirm that it captured the architecture you want."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To train the model, we pass in our unlabeled training data, our predicted probabilistic (float) labels from the label aggregator, a dev set X and Y if we want to checkpoint the best model seen so far as we train, and any other keyword arguments that you'd like to update in the config dict. For this synthetic problem, the trends in our data are fairly simple, so our model will begin to overfit very quickly. If we do begin to overfit and see our dev score go down, however, the best model seen at any epoch will be restored at the end of training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[1 epo]: TRAIN:[loss=0.510] VALID:[accuracy=0.935]\n",
      "Saving model at iteration 1 with best score 0.935\n",
      "[2 epo]: TRAIN:[loss=0.477] VALID:[accuracy=0.973]\n",
      "Saving model at iteration 2 with best score 0.973\n",
      "[3 epo]: TRAIN:[loss=0.473] VALID:[accuracy=0.964]\n",
      "[4 epo]: TRAIN:[loss=0.471] VALID:[accuracy=0.973]\n",
      "[5 epo]: TRAIN:[loss=0.469] VALID:[accuracy=0.976]\n",
      "Saving model at iteration 5 with best score 0.976\n",
      "Restoring best model from iteration 5 with score 0.976\n",
      "Finished Training\n",
      "Accuracy: 0.976\n",
      "        y=1    y=2   \n",
      " l=1    222    24    \n",
      " l=2     0     754   \n"
     ]
    }
   ],
   "source": [
    "end_model.train_model((Xs[0], Y_train_ps), valid_data=(Xs[1], Ys[1]), lr=0.01, l2=0.01, batch_size=256, \n",
    "                n_epochs=5, checkpoint_metric='accuracy', checkpoint_metric_mode='max')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Evaluate"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a trained EndModel, we can now evaluate performance on our held-out test set, observing a substantial boost in F1 score over using the LabelModel directly. Remember that the label model predicts Y from the labeling functions, L, whereas the end model predicts Y from the raw input, X."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Label Model:\n",
      "Accuracy: 0.869\n",
      "Precision: 0.747\n",
      "Recall: 0.707\n",
      "F1: 0.727\n",
      "        y=1    y=2   \n",
      " l=1    174    72    \n",
      " l=2    59     695   \n",
      "\n",
      "End Model:\n",
      "Accuracy: 0.965\n",
      "Precision: 1.000\n",
      "Recall: 0.858\n",
      "F1: 0.923\n",
      "        y=1    y=2   \n",
      " l=1    211    35    \n",
      " l=2     0     754   \n"
     ]
    }
   ],
   "source": [
    "print(\"Label Model:\")\n",
    "score = label_model.score((Ls[2], Ys[2]), metric=['accuracy','precision', 'recall', 'f1'])\n",
    "\n",
    "print()\n",
    "\n",
    "print(\"End Model:\")\n",
    "score = end_model.score((Xs[2], Ys[2]), metric=['accuracy','precision', 'recall', 'f1'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And there you go! If you'd like to learn about some of the other features in Snorkel MeTaL, including support for multi-task learning (MTL), hyperparameter tuning, and synthetic data generation, give some of the other tutorial notebooks a try!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python [conda env:metal]",
   "language": "python",
   "name": "conda-env-metal-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}