tutorials/crowdsourcing/Crowdsourced_Sentiment_Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Training a Sentiment Analysis LSTM Using Noisy Crowd Labels"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we'll provide a simple walkthrough of how to use Snorkel to resolve conflicts in a noisy crowdsourced dataset for a sentiment analysis task, and then use these denoised labels to train an LSTM sentiment analysis model which can be applied to new, unseen data to automatically make predictions!\n",
    "\n",
    "Specifically, we'll look at:\n",
    "1. Loading data via SparkSQL\n",
    "2. Creating basic Snorkel objects: `Candidates`, `Contexts`, and `Labels`\n",
    "3. Training the `GenerativeModel` to resolve labeling conflicts\n",
    "4. Training a simple LSTM sentiment analysis model, which can then be used on new, unseen data!\n",
    "\n",
    "Note that this is a simple tutorial meant to give an overview of the mechanics of using Snorkel-- we'll note places where more careful fine-tuning could be done!\n",
    "\n",
    "## Installing `PySpark`\n",
    "\n",
    "Please see the [official instructions](https://spark.apache.org/docs/latest/spark-standalone.html)!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Task Detail: Weather Sentiments in Tweets\n",
    "\n",
    "In this tutorial we focus on the [Weather sentiment](https://www.crowdflower.com/data/weather-sentiment/) task from [Crowdflower](https://www.crowdflower.com/).\n",
    "\n",
    "In this task, contributors were asked to grade the sentiment of a particular tweet relating to the weather. Contributors could choose among the following categories:\n",
    "1. Positive\n",
    "2. Negative\n",
    "3. I can't tell\n",
    "4. Neutral / author is just sharing information\n",
    "5. Tweet not related to weather condition\n",
    "\n",
    "The catch is that 20 contributors graded each tweet. Thus, in many cases contributors assigned conflicting sentiment labels to the same tweet. \n",
    "\n",
    "The task comes with two data files (to be found in the `data` directory of the tutorial:\n",
    "1. [weather-non-agg-DFE.csv](data/weather-non-agg-DFE.csv) contains the raw contributor answers for each of the 1,000 tweets.\n",
    "2. [weather-evaluated-agg-DFE.csv](data/weather-evaluated-agg-DFE.csv) contains gold sentiment labels by trusted workers for each of the 1,000 tweets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2\n",
    "%matplotlib inline\n",
    "import os\n",
    "import numpy as np\n",
    "from snorkel import SnorkelSession\n",
    "session = SnorkelSession()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1: Preprocessing - Data Loading with Spark SQL and Dataframes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we initialize a `SparkSession`, which manages a connection to a local Spark master which allows us to preprocess the raw data and prepare convert them to the necessary `Snorkel` format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Initialize Spark Environment and Spark SQL\n",
    "from pyspark.sql import SparkSession\n",
    "from pyspark.sql.functions import *\n",
    "from pyspark.sql.types import *\n",
    "from pyspark import SparkContext, SparkConf\n",
    "\n",
    "spark = SparkSession \\\n",
    "    .builder \\\n",
    "    .master(\"local\") \\\n",
    "    .appName(\"Snorkel Crowdsourcing Demo\") \\\n",
    "    .config(\"spark.some.config.option\", \"some-value\") \\\n",
    "    .getOrCreate()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now load the raw data for our crowdsourcing task (stored in a local csv file) into a dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "root\n",
      " |-- _unit_id_: string (nullable = true)\n",
      " |-- channel: string (nullable = true)\n",
      " |-- trust: string (nullable = true)\n",
      " |-- worker_id: string (nullable = true)\n",
      " |-- country: string (nullable = true)\n",
      " |-- region: string (nullable = true)\n",
      " |-- city: string (nullable = true)\n",
      " |-- emotion: string (nullable = true)\n",
      " |-- tweet_id: string (nullable = true)\n",
      " |-- tweet_body: string (nullable = true)\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# Load Raw Crowdsourcing Data\n",
    "raw_crowd_answers = spark.read.format(\"csv\").option(\"header\", \"true\").csv(\"data/weather-non-agg-DFE.csv\")\n",
    "raw_crowd_answers.printSchema()\n",
    "\n",
    "# Load Groundtruth Crowdsourcing Data\n",
    "gold_crowd_answers = spark.read.format(\"csv\").option(\"header\", \"true\").csv(\"data/weather-evaluated-agg-DFE.csv\")\n",
    "gold_crowd_answers.createOrReplaceTempView(\"gold_crowd_answers\")\n",
    "# Filter out low-confidence answers\n",
    "gold_answers = spark.sql(\"SELECT tweet_id, sentiment, tweet_body FROM gold_crowd_answers WHERE correct_category ='Yes' and correct_category_conf = 1\").orderBy(\"tweet_id\")\n",
    "\n",
    "# Keep Only the Tweets with Available Groundtruth\n",
    "candidate_labeled_tweets = raw_crowd_answers.join(gold_answers, raw_crowd_answers.tweet_id == gold_answers.tweet_id).select(raw_crowd_answers.tweet_id,raw_crowd_answers.tweet_body,raw_crowd_answers.worker_id,raw_crowd_answers.emotion)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As mentioned above, contributors can provide conflicting labels for the same tweet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "+---------+--------------------+--------------------+\n",
      "|worker_id|             emotion|          tweet_body|\n",
      "+---------+--------------------+--------------------+\n",
      "|  6498214|        I can't tell|I dunno which ass...|\n",
      "|  7450342|Neutral / author ...|I dunno which ass...|\n",
      "| 10752241|            Positive|I dunno which ass...|\n",
      "| 10235355|            Negative|I dunno which ass...|\n",
      "| 17475684|            Negative|I dunno which ass...|\n",
      "|  6346694|Neutral / author ...|I dunno which ass...|\n",
      "| 14806909|Neutral / author ...|I dunno which ass...|\n",
      "| 19028457|            Positive|I dunno which ass...|\n",
      "|  6737418|            Negative|I dunno which ass...|\n",
      "| 14584835|            Negative|I dunno which ass...|\n",
      "| 18381123|Neutral / author ...|I dunno which ass...|\n",
      "| 16498372|Tweet not related...|I dunno which ass...|\n",
      "|  7012325|            Positive|I dunno which ass...|\n",
      "|  9333400|            Negative|I dunno which ass...|\n",
      "| 10379699|            Positive|I dunno which ass...|\n",
      "| 14298198|            Positive|I dunno which ass...|\n",
      "| 20043586|            Negative|I dunno which ass...|\n",
      "|  9289735|        I can't tell|I dunno which ass...|\n",
      "| 16738677|            Negative|I dunno which ass...|\n",
      "| 15846764|            Negative|I dunno which ass...|\n",
      "+---------+--------------------+--------------------+\n",
      "only showing top 20 rows\n",
      "\n"
     ]
    }
   ],
   "source": [
    "candidate_labeled_tweets.select(\"worker_id\", \"emotion\", \"tweet_body\").orderBy(\"tweet_id\").show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2: Generating Snorkel Objects\n",
    "\n",
    "### `Candidates`\n",
    "\n",
    "`Candidates` are the core objects in Snorkel representing objects to be classified. We'll use a helper function to create a custom `Candidate` sub-class, `Tweet`, with values representing the possible labels that it can be classified with:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.models import candidate_subclass\n",
    "\n",
    "values = map(\n",
    "    lambda r: r.emotion,\n",
    "    candidate_labeled_tweets.select(\"emotion\").distinct().collect()\n",
    ")\n",
    "\n",
    "Tweet = candidate_subclass('Tweet', ['tweet'], values=values)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `Contexts`\n",
    "\n",
    "All `Candidate` objects point to one or more `Context` objects, which represent the raw data that they are rooted in. In this case, our candidates will each point to a single `Context` object representing the raw text of the tweet.\n",
    "\n",
    "Once we have defined the `Context` for each `Candidate`, we can commit them to the database. Note that we also split into two sets while doing this:\n",
    "\n",
    "1. **Training set (`split=0`):** The tweets for which we have noisy, conflicting crowd labels; we will resolve these conflicts using the `GenerativeModel` and then use them as training data for the LSTM\n",
    "\n",
    "2. **Test set (`split=1`):** We will pretend that we do not have any crowd labels for this split of the data, and use these to test the LSTM's performance on unseen data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from snorkel.models import Context, Candidate\n",
    "from snorkel.contrib.models.text import RawText\n",
    "\n",
    "# Make sure DB is cleared\n",
    "session.query(Context).delete()\n",
    "session.query(Candidate).delete()\n",
    "\n",
    "# Now we create the candidates with a simple loop\n",
    "tweet_bodies = candidate_labeled_tweets \\\n",
    "    .select(\"tweet_id\", \"tweet_body\") \\\n",
    "    .orderBy(\"tweet_id\") \\\n",
    "    .distinct()\n",
    "\n",
    "# Generate and store the tweet candidates to be classified\n",
    "# Note: We split the tweets in two sets: one for which the crowd \n",
    "# labels are not available to Snorkel (test, 10%) and one for which we assume\n",
    "# crowd labels are obtained (to be used for training, 90%)\n",
    "total_tweets = tweet_bodies.count()\n",
    "test_split = total_tweets*0.1\n",
    "for i, t in enumerate(tweet_bodies.collect()):\n",
    "    split = 1 if i <= test_split else 0\n",
    "    raw_text = RawText(stable_id=t.tweet_id, name=t.tweet_id, text=t.tweet_body)\n",
    "    tweet = Tweet(tweet=raw_text, split=split)\n",
    "    session.add(tweet)\n",
    "session.commit()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `Labels`\n",
    "\n",
    "Next, we'll store the labels for each of the training candidates in a sparse matrix (which will also automatically be saved to the Snorkel database), with one row for each candidate and one column for each crowd worker:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Clearing existing...\n",
      "Running UDF...\n",
      "[========================================] 100%%\n",
      "\n",
      "CPU times: user 4.36 s, sys: 98.3 ms, total: 4.46 s\n",
      "Wall time: 4.46 s\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<568x102 sparse matrix of type '<type 'numpy.int64'>'\n",
       "\twith 11360 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from snorkel.annotations import LabelAnnotator\n",
    "from collections import defaultdict\n",
    "\n",
    "# Extract worker votes\n",
    "# Cache locally to speed up for this small set\n",
    "worker_labels = candidate_labeled_tweets.select(\"tweet_id\", \"worker_id\", \"emotion\").collect()\n",
    "wls = defaultdict(list)\n",
    "for row in worker_labels:\n",
    "    wls[row.tweet_id].append((row.worker_id, row.emotion))\n",
    "\n",
    "# Create a label generator\n",
    "def worker_label_generator(t):\n",
    "    \"\"\"A generator over the different (worker_id, label_id) pairs for a Tweet.\"\"\"\n",
    "    for worker_id, label in wls[t.tweet.name]:\n",
    "        yield worker_id, label\n",
    "\n",
    "labeler = LabelAnnotator(label_generator=worker_label_generator)\n",
    "%time L_train = labeler.apply(split=0)\n",
    "L_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we load the ground truth (\"gold\") labels for both the training and test sets, and store as numpy arrays\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "gold_labels = defaultdict(list)\n",
    "\n",
    "# Get gold labels in verbose form\n",
    "verbose_labels = dict([(t.tweet_id, t.sentiment) \n",
    "                       for t in gold_answers.select(\"tweet_id\", \"sentiment\").collect()])\n",
    "\n",
    "# Iterate over splits, align with Candidate ordering\n",
    "for split in range(2):\n",
    "    cands = session.query(Tweet).filter(Tweet.split == split).order_by(Tweet.id).all() \n",
    "    for c in cands:\n",
    "        gold_labels[split].append(values.index(verbose_labels[c.tweet.name]) + 1)\n",
    "          \n",
    "train_cand_labels = np.array(gold_labels[0])\n",
    "test_cand_labels = np.array(gold_labels[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 3: Resolving Crowd Conflicts with the Generative Model\n",
    "\n",
    "Until now we have converted the raw crowdsourced data into a labeling matrix that can be provided as input to `Snorkel`. We will now show how to:\n",
    "\n",
    "1. Use `Snorkel's` generative model to learn the accuracy of each crowd contributor.\n",
    "2. Use the learned model to estimate a marginal distribution over the domain of possible labels for each task.\n",
    "3. Use the estimated marginal distribution to obtain the maximum a posteriori probability estimate for the label that each task takes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Imports\n",
    "from snorkel.learning.gen_learning import GenerativeModel\n",
    "\n",
    "# Initialize Snorkel's generative model for\n",
    "# learning the different worker accuracies.\n",
    "gen_model = GenerativeModel(lf_propensity=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Inferred cardinality: 5\n"
     ]
    }
   ],
   "source": [
    "# Train the generative model\n",
    "gen_model.train(\n",
    "    L_train,\n",
    "    reg_type=2,\n",
    "    reg_param=0.1,\n",
    "    epochs=30\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Infering the MAP assignment for each task\n",
    "Each task corresponds to an indipendent random variable. Thus, we can simply associate each task with the most probably label based on the estimated marginal distribution and get an accuracy score:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.996478873239\n",
      "Number incorrect: 2\n"
     ]
    }
   ],
   "source": [
    "correct, incorrect = gen_model.score(session, L_train, train_cand_labels)\n",
    "print \"Number incorrect:\", len(incorrect)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Majority vote\n",
    "\n",
    "It seems like we did well- but how well?  Given that this is a fairly simple task--we have 20 contributors per tweet (and most of them are far better than random)--**we expect majority voting to perform extremely well**, so we can check against majority vote:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.984154929577\n",
      "Number incorrect: 9\n"
     ]
    }
   ],
   "source": [
    "from collections import Counter\n",
    "\n",
    "# Collect the majority vote answer for each tweet\n",
    "mv = []\n",
    "for i in range(L_train.shape[0]):\n",
    "    c = Counter([L_train[i,j] for j in L_train[i].nonzero()[1]])\n",
    "    mv.append(c.most_common(1)[0][0])\n",
    "mv = np.array(mv)\n",
    "\n",
    "# Count the number correct by majority vote\n",
    "n_correct = np.sum([1 for i in range(L_train.shape[0]) if mv[i] == train_cand_labels[i]])\n",
    "print \"Accuracy:\", n_correct / float(L_train.shape[0])\n",
    "print \"Number incorrect:\", L_train.shape[0] - n_correct"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that while majority vote makes 9 errors, the Snorkel model makes only 2!  What about an average crowd worker?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Average human accuracy\n",
    "\n",
    "We see that the average accuracy of a single crowd worker is in fact much lower:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean Accuracy: 0.733649655207\n"
     ]
    }
   ],
   "source": [
    "accs = []\n",
    "for j in range(L_train.shape[1]):\n",
    "    n_correct = np.sum([1 for i in range(L_train.shape[0]) if L_train[i,j] == train_cand_labels[i]])\n",
    "    acc = n_correct / float(L_train[:,j].nnz)\n",
    "    accs.append(acc)\n",
    "print \"Mean Accuracy:\", np.mean(accs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4: Training an ML Model with Snorkel for Sentiment Analysis over Unseen Tweets\n",
    "\n",
    "In the previous step, we saw that Snorkel's generative model can help to denoise crowd labels automatically. However, what happens when we don't have noisy crowd labels for a tweet?\n",
    "\n",
    "In this step, we'll use the estimates of the generative model as _probabilistic training labels_ to train a simple LSTM sentiment analysis model, which takes as input a tweet **for which no crowd labels are available** and predicts its sentiment.\n",
    "\n",
    "First, we get the probabilistic training labels (_training marginals_) which are just the marginal estimates of the generative model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "train_marginals = gen_model.marginals(L_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Saved 568 marginals\n"
     ]
    }
   ],
   "source": [
    "from snorkel.annotations import save_marginals\n",
    "save_marginals(session, L_train, train_marginals)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we'll train a simple LSTM:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[textRNN] Dimension=100  LR=0.01\n",
      "[textRNN] Begin preprocessing\n",
      "[textRNN] Preprocessing done (2.25s)\n",
      "[textRNN] Training model\n",
      "[textRNN] #examples=568  #epochs=200  batch size=256\n",
      "[textRNN] Epoch 0 (0.93s)\tAverage loss=1.542936\n",
      "[textRNN] Epoch 5 (2.77s)\tAverage loss=0.208064\n",
      "[textRNN] Epoch 10 (4.61s)\tAverage loss=0.043410\n",
      "[textRNN] Epoch 15 (6.49s)\tAverage loss=0.033808\n",
      "[textRNN] Epoch 20 (8.47s)\tAverage loss=0.039784\n",
      "[textRNN] Epoch 25 (10.30s)\tAverage loss=0.032047\n",
      "[textRNN] Epoch 30 (12.13s)\tAverage loss=0.027176\n",
      "[textRNN] Epoch 35 (13.92s)\tAverage loss=0.026516\n",
      "[textRNN] Epoch 40 (15.68s)\tAverage loss=0.033868\n",
      "[textRNN] Epoch 45 (17.48s)\tAverage loss=0.028366\n",
      "[textRNN] Epoch 50 (19.36s)\tAverage loss=0.026088\n",
      "[textRNN] Epoch 55 (21.16s)\tAverage loss=0.028486\n",
      "[textRNN] Epoch 60 (23.14s)\tAverage loss=0.034369\n",
      "[textRNN] Epoch 65 (24.98s)\tAverage loss=0.026515\n",
      "[textRNN] Epoch 70 (26.80s)\tAverage loss=0.026159\n",
      "[textRNN] Epoch 75 (28.59s)\tAverage loss=0.025399\n",
      "[textRNN] Epoch 80 (30.42s)\tAverage loss=0.023513\n",
      "[textRNN] Epoch 85 (32.23s)\tAverage loss=0.021189\n",
      "[textRNN] Epoch 90 (34.19s)\tAverage loss=0.022252\n",
      "[textRNN] Epoch 95 (36.52s)\tAverage loss=0.023710\n",
      "[textRNN] Epoch 100 (38.36s)\tAverage loss=0.022882\n",
      "[textRNN] Epoch 105 (40.25s)\tAverage loss=0.022232\n",
      "[textRNN] Epoch 110 (42.00s)\tAverage loss=0.021887\n",
      "[textRNN] Epoch 115 (44.03s)\tAverage loss=0.023037\n",
      "[textRNN] Epoch 120 (45.88s)\tAverage loss=0.024601\n",
      "[textRNN] Epoch 125 (47.72s)\tAverage loss=0.019243\n",
      "[textRNN] Epoch 130 (49.43s)\tAverage loss=0.020191\n",
      "[textRNN] Epoch 135 (51.56s)\tAverage loss=0.020155\n",
      "[textRNN] Epoch 140 (53.50s)\tAverage loss=0.022676\n",
      "[textRNN] Epoch 145 (55.26s)\tAverage loss=0.019469\n",
      "[textRNN] Epoch 150 (57.12s)\tAverage loss=0.019763\n",
      "[textRNN] Epoch 155 (59.01s)\tAverage loss=0.019151\n",
      "[textRNN] Epoch 160 (60.82s)\tAverage loss=0.019512\n",
      "[textRNN] Epoch 165 (62.87s)\tAverage loss=0.018848\n",
      "[textRNN] Epoch 170 (64.92s)\tAverage loss=0.020663\n",
      "[textRNN] Epoch 175 (66.77s)\tAverage loss=0.019069\n",
      "[textRNN] Epoch 180 (68.77s)\tAverage loss=0.019311\n",
      "[textRNN] Epoch 185 (70.62s)\tAverage loss=0.019740\n",
      "[textRNN] Epoch 190 (72.82s)\tAverage loss=0.019249\n",
      "[textRNN] Epoch 195 (74.64s)\tAverage loss=0.019659\n",
      "[textRNN] Epoch 199 (76.00s)\tAverage loss=0.018687\n",
      "[textRNN] Training done (76.00s)\n"
     ]
    }
   ],
   "source": [
    "from snorkel.learning import TextRNN\n",
    "\n",
    "train_kwargs = {\n",
    "    'lr':         0.01,\n",
    "    'dim':        100,\n",
    "    'n_epochs':   200,\n",
    "    'dropout':    0.2,\n",
    "    'print_freq': 5\n",
    "}\n",
    "\n",
    "lstm = TextRNN(seed=1701, cardinality=Tweet.cardinality)\n",
    "train_cands = session.query(Tweet).filter(Tweet.split == 0).order_by(Tweet.id).all()\n",
    "lstm.train(train_cands, train_marginals, **train_kwargs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy: 0.703125\n",
      "Number incorrect: 19\n"
     ]
    }
   ],
   "source": [
    "test_cands = session.query(Tweet).filter(Tweet.split == 1).order_by(Tweet.id).all()\n",
    "correct, incorrect = lstm.score(session, test_cands, test_cand_labels)\n",
    "print \"Number incorrect:\", len(incorrect)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that we're already close to the accuracy of an average crowd worker! If we wanted to improve the score, we could tune the LSTM model using grid search (see the Intro tutorial), use [pre-trained word embeddings](https://nlp.stanford.edu/projects/glove/), or many other common techniques for getting state-of-the-art scores. Notably, we're doing this without using gold labels, but rather noisy crowd-labels!\n",
    "\n",
    "For more, checkout the other tutorials!"
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}