SocialMedia - Part 2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " <table><tr><td><img src=\"images/dbmi_logo.png\" width=\"75\" height=\"73\" alt=\"Pitt Biomedical Informatics logo\"></td><td><img src=\"images/pitt_logo.png\" width=\"75\" height=\"75\" alt=\"University of Pittsburgh logo\"></td></tr></table>\n",
    " \n",
    "\n",
    "# Social Media and Data Science - Part 2\n",
    "\n",
    "Data science modules developed by the University of Pittsburgh Biomedical Informatics Training Program with the support of the National Library of Medicine data science supplement to the University of Pittsburgh (Grant # T15LM007059-30S1). \n",
    "\n",
    "Developed by Harry Hochheiser, harryh@pitt.edu. All errors are my responsibility.\n",
    "\n",
    "<a rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc/4.0/\"><img alt=\"Creative Commons License\" style=\"border-width:0\" src=\"https://i.creativecommons.org/l/by-nc/4.0/88x31.png\" /></a><br />This work is licensed under a <a rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc/4.0/\">Creative Commons Attribution-NonCommercial 4.0 International License</a>.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  *Goal*: Use social media posts to explore the appplication of text and natural language processing to see what might be learned from online interactions.\n",
    "\n",
    "Specifically, we will retrieve, annotate, process, and interpret Twitter data on health-related issues such as depression."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--- \n",
    "References:\n",
    "* [Mining Twitter Data with Python (Part 1: Collecting data)](https://marcobonzanini.com/2015/03/02/mining-twitter-data-with-python-part-1/)\n",
    "* The [Tweepy Python API for Twitter](http://www.tweepy.org/)\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import numpy as np\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import jsonpickle\n",
    "import json\n",
    "import random\n",
    "import tweepy\n",
    "import time\n",
    "import operator\n",
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.0.1 Setup\n",
    "\n",
    "Before we dig in, we must grab a bit of code from [Part 1](SocialMedia%20-%20Part%201.ipynb):\n",
    "\n",
    "1. The Tweets class used to store the tweets.\n",
    "2. Our twitter API Keys - be sure to copy the keys that you generated when you completed [Part 1](SocialMedia%20-%20Part%201.ipynb).\n",
    "3. Configuration of our Twitter connection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Tweets:\n",
    "    \n",
    "    \n",
    "    def __init__(self,term=\"\",corpus_size=100):\n",
    "        self.tweets={}\n",
    "        if term !=\"\":\n",
    "            self.searchTwitter(term,corpus_size)\n",
    "    \n",
    "    def searchTwitter(self,term,corpus_size):\n",
    "        searchTime=datetime.now()\n",
    "        while (self.countTweets() < corpus_size):\n",
    "            new_tweets = api.search(term,lang=\"en\",tweet_mode='extended',count=corpus_size)\n",
    "            for nt_json in new_tweets:\n",
    "                nt = nt_json._json\n",
    "                if self.getTweet(nt['id_str']) is None and self.countTweets() < corpus_size:\n",
    "                    self.addTweet(nt,searchTime,term)\n",
    "            time.sleep(120)\n",
    "                \n",
    "    def addTweet(self,tweet,searchTime,term=\"\",count=0):\n",
    "        id = tweet['id_str']\n",
    "        if id not in self.tweets.keys():\n",
    "            self.tweets[id]={}\n",
    "            self.tweets[id]['tweet']=tweet\n",
    "            self.tweets[id]['count']=0\n",
    "            self.tweets[id]['searchTime']=searchTime\n",
    "            self.tweets[id]['searchTerm']=term\n",
    "        self.tweets[id]['count'] = self.tweets[id]['count'] +1\n",
    "        \n",
    "    def combineTweets(self,other):\n",
    "        for otherid in other.getIds():\n",
    "            tweet = other.getTweet(otherid)\n",
    "            searchTerm = other.getSearchTerm(otherid)\n",
    "            searchTime = other.getSearchTime(otherid)\n",
    "            self.addTweet(tweet,searchTime,searchTerm)\n",
    "        \n",
    "    def getTweet(self,id):\n",
    "        if id in self.tweets:\n",
    "            return self.tweets[id]['tweet']\n",
    "        else:\n",
    "            return None\n",
    "    \n",
    "    def getTweetCount(self,id):\n",
    "        return self.tweets[id]['count']\n",
    "    \n",
    "    def countTweets(self):\n",
    "        return len(self.tweets)\n",
    "    \n",
    "    # return a sorted list of tupes of the form (id,count), with the occurrence counts sorted in decreasing order\n",
    "    def mostFrequent(self):\n",
    "        ps = []\n",
    "        for t,entry in self.tweets.items():\n",
    "            count = entry['count']\n",
    "            ps.append((t,count))  \n",
    "        ps.sort(key=lambda x: x[1],reverse=True)\n",
    "        return ps\n",
    "    \n",
    "    # reeturns tweet IDs as a set\n",
    "    def getIds(self):\n",
    "        return set(self.tweets.keys())\n",
    "    \n",
    "    # save the tweets to a file\n",
    "    def saveTweets(self,filename):\n",
    "        json_data =jsonpickle.encode(self.tweets)\n",
    "        with open(filename,'w') as f:\n",
    "            json.dump(json_data,f)\n",
    "    \n",
    "    # read the tweets from a file \n",
    "    def readTweets(self,filename):\n",
    "        with open(filename,'r') as f:\n",
    "            json_data = json.load(f)\n",
    "            incontents = jsonpickle.decode(json_data)   \n",
    "            self.tweets=incontents\n",
    "        \n",
    "    def getSearchTerm(self,id):\n",
    "        return self.tweets[id]['searchTerm']\n",
    "    \n",
    "    def getSearchTime(self,id):\n",
    "        return self.tweets[id]['searchTime']\n",
    "    \n",
    "    def getText(self,id):\n",
    "        tweet = self.getTweet(id)\n",
    "        text=tweet['full_text']\n",
    "        if 'retweeted_status'in tweet:\n",
    "            original = tweet['retweeted_status']\n",
    "            text=original['full_text']\n",
    "        return text"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Put the values of your keys into these variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "consumer_key = 'YOUR-CONSUMER-KEY'\n",
    "consumer_secret = 'YOUR-CONSUMER-SECRET'\n",
    "access_token = 'YOUR-ACCESS-TOKEN'\n",
    "access_secret = 'YOUR-ACCESS-SECRET'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tweepy import OAuthHandler\n",
    "\n",
    "auth = OAuthHandler(consumer_key, consumer_secret)\n",
    "auth.set_access_token(access_token, access_secret)\n",
    "\n",
    "api = tweepy.API(auth)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2.1 Annotating Tweets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have a corpus of tweets, what do we want to do with them? Turning a relatively vague notion into a well-defined research question is often a significant challenge, as examination of the data often reveals both shortcomings and unforeseen opportunities.\n",
    "\n",
    "In our case, we are interested in looking at tweets about smoking, but we're not quite sure exactly *what* we are looking for. We have a vague notion that we might learn something interesting, but understanding exactly what that is, and what sort of analyses we might need, will require a bit more work.\n",
    "\n",
    "In situations such as this, we might look at some of the data to form some preliminary impressions of the content. Specifically, we can look at indidividual tweets, assigning them to one or more categories - known as *codes* - based on their content.  We can add categories as needed to capture important ideas that we might want to refer back to. This practice - known as *open coding* allows us to begin to make sense of unfamiliar data sets. \n",
    "\n",
    "This sounds much more complicated than it is. For now, let's start with some of the data that you used  in [Part 1](SocialMedia%20-%20Part%201.ipynb). If you completed this exercise, you should have 100 tweets associated with the search term 'smoking' in a file names `tweets.json`. If you didn't complete this exercise, please go back and do so."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets =Tweets()\n",
    "tweets.readTweets(\"tweets.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We check the count, to verify the contents..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(tweets.countTweets())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will begin by taking a look at a subset of the first 20 tweets\n",
    "\n",
    "To get this list, we'll sort the ids of the tweets and take the first 10 in the list, as ordered by ID"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ids=list(tweets.getIds())\n",
    "ids.sort()\n",
    "working=[]\n",
    "for i in range(20):\n",
    "    id = ids[i]\n",
    "    working.append(id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*working* now has 20 tweets ids. Let's start with the first."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "td = working[0]\n",
    "print(tweets.getSearchTerm(id))\n",
    "print(tweets.getSearchTime(id))\n",
    "print(tweets.getText(td))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tweet might have any of several interesting characteristics. It might be a retweet; it might specifically mention marijunana or tobacco; or it might not be related to either of these. \n",
    "\n",
    "We can model any and all of these points through relevant annotation. Specifically, we will a new array of codes to each tweet object. This array will contain a list of categorical annotations associated with the tweet.  We add routines to add a single code to a tweet (by ID), to add multiple codes, and to retrieve the list of codes associated with a tweet.\n",
    "\n",
    "\n",
    "See modifications to the  Tweets object in this new definition. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Tweets:\n",
    "    \n",
    "    \n",
    "    def __init__(self,term=\"\",corpus_size=100):\n",
    "        self.tweets={}\n",
    "        if term !=\"\":\n",
    "            self.searchTwitter(term,corpus_size)\n",
    "                \n",
    "    def searchTwitter(self,term,corpus_size):\n",
    "        searchTime=datetime.now()\n",
    "        while (self.countTweets() < corpus_size):\n",
    "            new_tweets = api.search(term,lang=\"en\",tweet_mode='extended',count=corpus_size)\n",
    "            for nt_json in new_tweets:\n",
    "                nt = nt_json._json\n",
    "                if self.getTweet(nt['id_str']) is None and self.countTweets() < corpus_size:\n",
    "                    self.addTweet(nt,searchTime,term)\n",
    "            time.sleep(120)\n",
    "                \n",
    "    def addTweet(self,tweet,searchTime,term=\"\",count=0):\n",
    "        id = tweet['id_str']\n",
    "        if id not in self.tweets.keys():\n",
    "            self.tweets[id]={}\n",
    "            self.tweets[id]['tweet']=tweet\n",
    "            self.tweets[id]['count']=0\n",
    "            self.tweets[id]['searchTime']=searchTime\n",
    "            self.tweets[id]['searchTerm']=term\n",
    "        self.tweets[id]['count'] = self.tweets[id]['count'] +1\n",
    "\n",
    "    def combineTweets(self,other):\n",
    "        for otherid in other.getIds():\n",
    "            tweet = other.getTweet(otherid)\n",
    "            searchTerm = other.getSearchTerm(otherid)\n",
    "            searchTime = other.getSearchTime(otherid)\n",
    "            self.addTweet(tweet,searchTime,searchTerm)\n",
    "        \n",
    "    def getTweet(self,id):\n",
    "        if id in self.tweets:\n",
    "            return self.tweets[id]['tweet']\n",
    "        else:\n",
    "            return None\n",
    "    \n",
    "    def getTweetCount(self,id):\n",
    "        return self.tweets[id]['count']\n",
    "    \n",
    "    def countTweets(self):\n",
    "        return len(self.tweets)\n",
    "    \n",
    "    # return a sorted list of tupes of the form (id,count), with the occurrence counts sorted in decreasing order\n",
    "    def mostFrequent(self):\n",
    "        ps = []\n",
    "        for t,entry in self.tweets.items():\n",
    "            count = entry['count']\n",
    "            ps.append((t,count))  \n",
    "        ps.sort(key=lambda x: x[1],reverse=True)\n",
    "        return ps\n",
    "    \n",
    "    # reeturns tweet IDs as a set\n",
    "    def getIds(self):\n",
    "        return set(self.tweets.keys())\n",
    "    \n",
    "    # save the tweets to a file\n",
    "    def saveTweets(self,filename):\n",
    "        json_data =jsonpickle.encode(self.tweets)\n",
    "        with open(filename,'w') as f:\n",
    "            json.dump(json_data,f)\n",
    "    \n",
    "    # read the tweets from a file \n",
    "    def readTweets(self,filename):\n",
    "        with open(filename,'r') as f:\n",
    "            json_data = json.load(f)\n",
    "            incontents = jsonpickle.decode(json_data)   \n",
    "            self.tweets=incontents\n",
    "        \n",
    "    def getSearchTerm(self,id):\n",
    "        return self.tweets[id]['searchTerm']\n",
    "    \n",
    "    def getSearchTime(self,id):\n",
    "        return self.tweets[id]['searchTime']\n",
    "    \n",
    "    def getText(self,id):\n",
    "        tweet = self.getTweet(id)\n",
    "        text=tweet['full_text']\n",
    "        if 'retweeted_status'in tweet:\n",
    "            original = tweet['retweeted_status']\n",
    "            text=original['full_text']\n",
    "        return text\n",
    "                \n",
    "    ### NEW ROUTINE - add a code to a tweet\n",
    "    def addCode(self,id,code):\n",
    "        tweet=self.getTweet(id)\n",
    "        if 'codes' not in tweet:\n",
    "            tweet['codes']=set()\n",
    "        tweet['codes'].add(code)\n",
    "        \n",
    "    ### NEW ROUTINE  - add multiple  codes for a tweet\n",
    "    def addCodes(self,id,codes):\n",
    "        for code in codes:\n",
    "            self.addCode(id,code)\n",
    "        \n",
    "    ### NEW ROUTINE get codes for a tweet\n",
    "    def getCodes(self,id):\n",
    "        tweet=self.getTweet(id)\n",
    "        if 'codes' in tweet:\n",
    "            return tweet['codes']\n",
    "        else:\n",
    "            return None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have this set up, we can load tweets from a file and reload the subset.\n",
    "\n",
    "Note that thsis file is not acutally real tweets - it's just fabricated tweets that have been put together for purposes of this lesson. The text, user name, and other details are all wrong or omitted.  Later, you'll try this with the tweets you loaded in your search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets =Tweets()\n",
    "tweets.readTweets(\"tweets-fake.json\")\n",
    "ids=list(tweets.getIds())\n",
    "ids.sort()\n",
    "working=[]\n",
    "\n",
    "for i in range(len(ids)):\n",
    "    id = ids[i]\n",
    "    working.append(id)\n",
    "\n",
    "td = working[0]\n",
    "t = tweets.getTweet(td)\n",
    "tweets.getText(td)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "so, this tweet seems to be about marijuana. We can add a code to this effect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets.addCode(td,\"MARIJUANA\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also confirm that this tweet is associated with the desired codes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets.getCodes(td)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Good. Let's look at the next tweet. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "td = working[1]\n",
    "tweets.getText(td)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tweet contains a generic mention of smoking, without much detail, but we might gues that it is also about marijuana"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets.addCodes(td,['MARIJUANA'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ok.. moving on to the third tweet.."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "td = working[2]\n",
    "tweets.getText(td)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets.addCodes(td,['QUITTING','BENEFITS'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "next..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "td = working[3]\n",
    "tweets.getText(td)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This note mentions government quitting efforts, and has both a link and a user mention."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets.addCodes(td,['LINK','USERMENTION','GOVERNMENT','QUITTING'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "td = working[4]\n",
    "tweets.getText(td)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This tweet mentions a negatigve attitude towards vaping."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets.addCodes(td,['VAPING','NEGATIVE'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've gone through several tweets, we can review the codes used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in range(5):\n",
    "    td=working[i]\n",
    "    print(tweets.getCodes(td))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Having annotated several tweets, we might want to save the annotations in a file for future use. Fortnuately, the approach that we've used in our save and reload code is flexible enough to handle this without any further changes to the implementation. \n",
    "\n",
    "How does this work? The `Tweets` class stores all of the information abou the tweets in a simple dictionary. Tweet counts and codes are then stored inside the tweet object. When we go to save the set of Tweets, we simply turn this dictionary into JSON and then write it to a file. To read things in, we just read the JSON from the file and convert the result back into a dictionary. Thus, anything that we add to the dictionary will automatically be writen out and read back in.  We still need additional routines to access this data (like `addCode`, `addCodes`, and `getCodes`), but we  don't need to change the save/load routines.  Let's try it out.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets.saveTweets(\"tweets-fake-annotated.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweets2=Tweets()\n",
    "tweets2.readTweets(\"tweets-fake-annotated.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print(tweets.getText(td))\n",
    "print(tweets.getCodes(td))\n",
    "print(tweets2.getText(td))\n",
    "print(tweets2.getCodes(td))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "****\n",
    "# Exercise 2.1: Examining the distribution of codes across a corpus\n",
    "\n",
    "Having annotated a number of tweets, you might want to get an idea of how many tweets are being used and how often. Write a new method inside the `Tweets` class to provide a list of codes and counts, sorted by frequency.  Please be sure to reload the tweets after you redefine the class.\n",
    "\n",
    "*ANSWER FOLLOWS - insert here*\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*END CUT HERE*\n",
    "****"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 2.2: Code the Next 10 tweets in the set. \n",
    "Start with the tags used above, adding your own as needed.  Code up to and including the tweet  with index 20 in the `working` array. Examine the code profile and save your tweets  to a new file when you are done. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*ANSWER FOLLOWS - insert here*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*END CUT*\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# EXERCISE 2.3: Reflection on coding\n",
    "\n",
    "Open coding can often be an iterative process. When we first start out, we don't really know what we're looking for. As a result, the first few items annotated might only get a few codes, and we might miss ideas that we don't initially think are important. As we see more and more items, our ideas of what needs to be annotated will change, and we'll start adding in codes that might also apply to earlier messages. Thus, we often need to review and re-annotate earlier tweets to account for changes in our interpreations.\n",
    "\n",
    "Review your annotations the tweets that you reviewed. Revise the codes associated with these tweets, adding items from the overall list of codes as appropriate. Describe the change that you have made.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "*ANSWER FOLLOWS - insert here*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "*END CUT*\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# EXERCISE 2.4: Reflection on storage/serialization\n",
    "\n",
    "In working with this small set of 100 tweets, we are taking a very simple approach to storage and management of the tweets and annotations. Storing everything in a nested Python dictionary and then dumping it to disk as JSON text can be very appealing. What are the strengths and weaknesses of this approach, and how might these strengths and weaknesses differ with larger datasets containing 100,000 or 100 million datasets? What alternative  strategies might you use for larger datasets?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "*ANSWER FOLLOWS - insert here*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "*END CUT*\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2.2 Final Notes\n",
    "\n",
    "[Part 3](SocialMedia%20-%20Part%203.ipynb) will explore the application of Natural Language Processing  - NLP - techniques to Tweet data. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6",
   "language": "python",
   "name": "python3.6"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}