SocialMedia - Part 0.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " <table><tr><td><img src=\"images/dbmi_logo.png\" width=\"75\" height=\"73\" alt=\"Pitt Biomedical Informatics logo\"></td><td><img src=\"images/pitt_logo.png\" width=\"75\" height=\"75\" alt=\"University of Pittsburgh logo\"></td></tr></table>\n",
    " \n",
    " \n",
    " # Social Media and Data Science - Part 0\n",
    " \n",
    " \n",
    "Data science modules developed by the University of Pittsburgh Biomedical Informatics Training Program with the support of the National Library of Medicine data science supplement to the University of Pittsburgh (Grant # T15LM007059-30S1). \n",
    "\n",
    "Developed by Harry Hochheiser, harryh@pitt.edu. All errors are my responsibility.\n",
    "\n",
    "<a rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc/4.0/\"><img alt=\"Creative Commons License\" style=\"border-width:0\" src=\"https://i.creativecommons.org/l/by-nc/4.0/88x31.png\" /></a><br />This work is licensed under a <a rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc/4.0/\">Creative Commons Attribution-NonCommercial 4.0 International License</a>.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  *Goal*: Learn how to retrieve, manage, and save social media posts.\n",
    "\n",
    "Specifically, we will retrieve, annotate, process, and interpret Twitter data on health-related issues such as smoking."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--- \n",
    "References:\n",
    "* [Mining Twitter Data with Python (Part 1: Collecting data)](https://marcobonzanini.com/2015/03/02/mining-twitter-data-with-python-part-1/)\n",
    "* The [Tweepy Python API for Twitter](http://www.tweepy.org/)\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 1. Introduction\n",
    "\n",
    "Analysis of social-media discussions has grown to be an important tool for biomedical informatics researchers, particularly for addressing questions relevant to public perceptions of health and related matters. Studies have examination of a range of topics at the intersection of health and social media, including studies of how [Facebook might be used to commuication health information](http://www.jmir.org/2016/8/e218/) how Tweets might be used to understand how smokers perceive [e-cigarettes, hookahs and other emerging smoking products](https://www.jmir.org/2013/8/e174/), and many others.\n",
    "\n",
    "Although each investigation has unique aspects, studies of social media generally share several common tasks. Data acquisition is often the first challenge: although some data may be freely available, there are often [limits](https://dev.twitter.com/rest/public/rate-limits) as to how much data can be queried easily. Researchers might look out for [opportunities for accessing larger amounts of data](https://www.wired.com/2014/02/twitter-promises-share-secrets-academia/). Some studies contract with [commercial services providing fee-based access](https://gnip.com). \n",
    "\n",
    "Once a data set is in hand, the next step is often to identify key terms and phrases relating to the research question. Messages might be annotated to indicate specific categorizations of interest - indicating, for example, if a message referred to a certain aspect of a disease or symptom. Similarly, key words and phrases regularly occurring in the content might also be identified. Natural language and text processing techniques might be used to extract key words, phrases, and relationships, and machine learning tools might be used to build classifiers capable of distinguishing between types of tweets of interest. \n",
    "\n",
    "This module presents a preliminary overview of these techniques, using Python 3 and several auxiliary libraries to explore the application of these techniques to Twitter data. \n",
    "\n",
    "[Part 1](SocialMedia%20-%20Part%201.ipynb) will cover the basics of retrieving data\n",
    "  \n",
    "  1. Configuration of tools to access Twitter data\n",
    "  2. Twitter data retrieval\n",
    "  3. Searching for tweets\n",
    "  \n",
    "[Part 2](SocialMedia%20-%20Part%202.ipynb) will cover annotation.\n",
    "\n",
    "[Part 3](SocialMedia%20-%20Part%203.ipynb) will discuss analysis through natural language processing and machine learning.\n",
    "\n",
    "[Part 4](SocialMedia%20-%20Part%204.ipynb) introduces basic classifiers and suggests how they might be used to classify tweets. Evaluation of classifiers is also discussed.\n",
    "\n",
    "[Part 5](SocialMedia%20-%20Part%204.ipynb) provides an exercise that ties all of this material togethers.\n",
    "\n",
    "Our case study will apply these topics to Twitter discussions of smoking and vaping. Although details of the tools used to access data and the format and content of the data may differ for various services, the strategies and procedures used to analyze the data will generalize to other tools.\n",
    "\n",
    "\n",
    "This doucment details the technical requirements for these lecutres on Social Media Data Science. Content on this page is intended to inform those who are involved in either (a) configuring Jupyter instances for running these notebooks, or (b) teaching or adapting this material.\n",
    "\n",
    "Others can jump right in with [Part 1](SocialMedia%20-%20Part%201.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 2. Software Dependencies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All components of these exercises run on the [Python](https://www.python.org) programming language. Python 3.6.5 was used in the develompent and testing of these exercises. As of October 2018, Python 3.6 is the best version to use, as some of the libraries listed below do not (yet) work with Python 3.7.\n",
    "\n",
    "These modules are presented as [Jupyter](https://jupyter.org), notebbooks. \n",
    "\n",
    "Python libraries used in these notebooks. You may need to install these libraries if you are creating a new computational environment: \n",
    "\n",
    "* [NumPy](http://www.numpy.org) - for preparing data for plotting\n",
    "* [Matplotlib](https://matplotlib.org) - plots and garphs\n",
    "* [jsonpickle](https://jsonpickle.github.io) for storing tweets. \n",
    "* [spaCy](https://spaCy.io/) - an NLP toolkit.\n",
    "* [scikit-learn](http://scikit-learn.org) for machine learning\n",
    "* [tweepy](http://www.tweepy.org) for retrieving Tweets via the Twitter API.\n",
    "\n",
    "If your Python installtaion is based on [pip](https://pip.pypa.io/en/stable/), you can run the instructions in the cell below to install these components if needed:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: numpy in c:\\users\\jebat\\anaconda3\\lib\\site-packages (1.14.3)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "thinc 6.10.3 requires msgpack<1.0.0,>=0.5.6, which is not installed.\n",
      "spacy 2.0.11 requires pathlib, which is not installed.\n",
      "msgpack-numpy 0.4.4.1 requires msgpack>=0.5.2, which is not installed.\n",
      "distributed 1.21.8 requires msgpack, which is not installed.\n",
      "spacy 2.0.11 has requirement regex==2017.4.5, but you'll have regex 2017.11.9 which is incompatible.\n",
      "You are using pip version 10.0.1, however version 18.1 is available.\n",
      "You should consider upgrading via the 'python -m pip install --upgrade pip' command.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting maplotlib\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "  Could not find a version that satisfies the requirement maplotlib (from versions: )\n",
      "No matching distribution found for maplotlib\n",
      "You are using pip version 10.0.1, however version 18.1 is available.\n",
      "You should consider upgrading via the 'python -m pip install --upgrade pip' command.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: jsonpickle in c:\\users\\jebat\\anaconda3\\lib\\site-packages\\jsonpickle-0.9.2-py2.7.egg (0.9.2)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "thinc 6.10.3 requires msgpack<1.0.0,>=0.5.6, which is not installed.\n",
      "spacy 2.0.11 requires pathlib, which is not installed.\n",
      "msgpack-numpy 0.4.4.1 requires msgpack>=0.5.2, which is not installed.\n",
      "distributed 1.21.8 requires msgpack, which is not installed.\n",
      "spacy 2.0.11 has requirement regex==2017.4.5, but you'll have regex 2017.11.9 which is incompatible.\n",
      "You are using pip version 10.0.1, however version 18.1 is available.\n",
      "You should consider upgrading via the 'python -m pip install --upgrade pip' command.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: spacy in c:\\users\\jebat\\anaconda3\\lib\\site-packages (2.0.11)\n",
      "Requirement already satisfied: numpy>=1.7 in c:\\users\\jebat\\anaconda3\\lib\\site-packages (from spacy) (1.14.3)\n",
      "Requirement already satisfied: murmurhash<0.29,>=0.28 in c:\\users\\jebat\\anaconda3\\lib\\site-packages (from spacy) (0.28.0)\n",
      "Requirement already satisfied: cymem<1.32,>=1.30 in c:\\users\\jebat\\anaconda3\\lib\\site-packages (from spacy) (1.31.2)\n",
      "Requirement already satisfied: preshed<2.0.0,>=1.0.0 in c:\\users\\jebat\\anaconda3\\lib\\site-packages (from spacy) (1.0.0)\n",
      "Requirement already satisfied: thinc<6.11.0,>=6.10.1 in c:\\users\\jebat\\anaconda3\\lib\\site-packages (from spacy) (6.10.3)\n",
      "Requirement already satisfied: plac<1.0.0,>=0.9.6 in c:\\users\\jebat\\anaconda3\\lib\\site-packages (from spacy) (0.9.6)\n",
      "Collecting pathlib (from spacy)\n",
      "  Downloading https://files.pythonhosted.org/packages/ac/aa/9b065a76b9af472437a0059f77e8f962fe350438b927cb80184c32f075eb/pathlib-1.0.1.tar.gz (49kB)\n",
      "Requirement already satisfied: ujson>=1.35 in c:\\users\\jebat\\anaconda3\\lib\\site-packages (from spacy) (1.35)\n",
      "Requirement already satisfied: dill<0.3,>=0.2 in c:\\users\\jebat\\anaconda3\\lib\\site-packages (from spacy) (0.2.8.2)\n",
      "Collecting regex==2017.4.5 (from spacy)\n",
      "  Downloading https://files.pythonhosted.org/packages/ad/0b/c1c5781a707e6ea01bcf57d8ad3c42125260fca67ef79206ecaef04a8754/regex-2017.04.05-cp36-none-win_amd64.whl (243kB)\n",
      "Collecting msgpack<1.0.0,>=0.5.6 (from thinc<6.11.0,>=6.10.1->spacy)\n",
      "  Downloading https://files.pythonhosted.org/packages/04/81/c6363198f24ec1c56e5c48ce685cb532e175125adade0cdb181c8c5fea6e/msgpack-0.5.6-cp36-cp36m-win_amd64.whl (85kB)\n",
      "Requirement already satisfied: msgpack-numpy<1.0.0,>=0.4.1 in c:\\users\\jebat\\anaconda3\\lib\\site-packages (from thinc<6.11.0,>=6.10.1->spacy) (0.4.4.1)\n",
      "Requirement already satisfied: cytoolz<0.10,>=0.9.0 in c:\\users\\jebat\\anaconda3\\lib\\site-packages (from thinc<6.11.0,>=6.10.1->spacy) (0.9.0.1)\n",
      "Requirement already satisfied: wrapt<1.11.0,>=1.10.0 in c:\\users\\jebat\\anaconda3\\lib\\site-packages (from thinc<6.11.0,>=6.10.1->spacy) (1.10.11)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.10.0 in c:\\users\\jebat\\anaconda3\\lib\\site-packages (from thinc<6.11.0,>=6.10.1->spacy) (4.26.0)\n",
      "Requirement already satisfied: six<2.0.0,>=1.10.0 in c:\\users\\jebat\\anaconda3\\lib\\site-packages (from thinc<6.11.0,>=6.10.1->spacy) (1.11.0)\n",
      "Requirement already satisfied: pyreadline>=1.7.1 in c:\\users\\jebat\\anaconda3\\lib\\site-packages (from dill<0.3,>=0.2->spacy) (2.1)\n",
      "Requirement already satisfied: toolz>=0.8.0 in c:\\users\\jebat\\anaconda3\\lib\\site-packages (from cytoolz<0.10,>=0.9.0->thinc<6.11.0,>=6.10.1->spacy) (0.9.0)\n",
      "Building wheels for collected packages: pathlib\n",
      "  Running setup.py bdist_wheel for pathlib: started\n",
      "  Running setup.py bdist_wheel for pathlib: finished with status 'done'\n",
      "  Stored in directory: C:\\Users\\jebat\\AppData\\Local\\pip\\Cache\\wheels\\f9\\b2\\4a\\68efdfe5093638a9918bd1bb734af625526e849487200aa171\n",
      "Successfully built pathlib\n",
      "Installing collected packages: pathlib, regex, msgpack\n",
      "  Found existing installation: regex 2017.11.9\n",
      "    Uninstalling regex-2017.11.9:\n",
      "      Successfully uninstalled regex-2017.11.9\n",
      "Successfully installed msgpack-0.5.6 pathlib-1.0.1 regex-2017.4.5\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You are using pip version 10.0.1, however version 18.1 is available.\n",
      "You should consider upgrading via the 'python -m pip install --upgrade pip' command.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: scikit-learn in c:\\users\\jebat\\anaconda3\\lib\\site-packages (0.19.1)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You are using pip version 10.0.1, however version 18.1 is available.\n",
      "You should consider upgrading via the 'python -m pip install --upgrade pip' command.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: tweepy in c:\\users\\jebat\\anaconda3\\lib\\site-packages (3.6.0)\n",
      "Requirement already satisfied: requests>=2.11.1 in c:\\users\\jebat\\anaconda3\\lib\\site-packages (from tweepy) (2.18.4)\n",
      "Requirement already satisfied: requests_oauthlib>=0.7.0 in c:\\users\\jebat\\anaconda3\\lib\\site-packages (from tweepy) (1.0.0)\n",
      "Requirement already satisfied: six>=1.10.0 in c:\\users\\jebat\\anaconda3\\lib\\site-packages (from tweepy) (1.11.0)\n",
      "Requirement already satisfied: PySocks>=1.5.7 in c:\\users\\jebat\\anaconda3\\lib\\site-packages (from tweepy) (1.6.8)\n",
      "Requirement already satisfied: chardet<3.1.0,>=3.0.2 in c:\\users\\jebat\\anaconda3\\lib\\site-packages (from requests>=2.11.1->tweepy) (3.0.4)\n",
      "Requirement already satisfied: idna<2.7,>=2.5 in c:\\users\\jebat\\anaconda3\\lib\\site-packages (from requests>=2.11.1->tweepy) (2.6)\n",
      "Requirement already satisfied: urllib3<1.23,>=1.21.1 in c:\\users\\jebat\\anaconda3\\lib\\site-packages (from requests>=2.11.1->tweepy) (1.22)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in c:\\users\\jebat\\anaconda3\\lib\\site-packages (from requests>=2.11.1->tweepy) (2018.4.16)\n",
      "Requirement already satisfied: oauthlib>=0.6.2 in c:\\users\\jebat\\anaconda3\\lib\\site-packages (from requests_oauthlib>=0.7.0->tweepy) (2.1.0)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "You are using pip version 10.0.1, however version 18.1 is available.\n",
      "You should consider upgrading via the 'python -m pip install --upgrade pip' command.\n"
     ]
    }
   ],
   "source": [
    "!pip install numpy\n",
    "!pip install maplotlib\n",
    "!pip install jsonpickle\n",
    "!pip install spacy\n",
    "!pip install scikit-learn\n",
    "!pip install tweepy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other installations, including those using [conda](https://conda.io/docs/) and related variants, might have slightly different installation instructions. Please contact your local systems administrator if you need help in configuring the libraries."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3. Case study description"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1 Learning Outcomes\n",
    "\n",
    "Upon completion of this module, students will be able to:\n",
    "\n",
    "* Understand the use of Application Programming Interfaces (APIs) to retrieve data from sites such as Twitter.\n",
    "* Understand the structure and content of resulting  data\n",
    "* Use and extend a Python class definition for managing extracted social media data, using Twitter as an example\n",
    "* Explore resulting social media data for patterns of authorship and other metadata.\n",
    "* Annotate/classify social media posts for further analysis.\n",
    "* Identify and discuss basic Natural Language Processing steps, including tokenization, lemmatization, part-of-speech tagging, and named entity recognition.\n",
    "* Use and extend code for executing key natural language processing pipeline steps.\n",
    "* Appreciate the relevance of vectorization for machine-learning classification of texts. \n",
    "* Convert tweets into appropriate vector representations.\n",
    "* Verify the ouptut of a vectorizer.\n",
    "* Divide a dataset into test and train sets for machine learning.\n",
    "* Verify the distribution of classes into test and train sets.\n",
    "* Train and evaluate an SVM-based classifier."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.2 Licensing/Restrictions/Access\n",
    "\n",
    "This work is licensed under a [Creative Commons Attribution-NonCommercial 4.0 International License](http://creativecommons.org/licenses/by-nc/4.0/\")."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.3 Target Student Audience\n",
    "\n",
    "Upper undergraduate or first-year graduate students."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.4 Prerequisite Skills and Knowledge Required\n",
    "\n",
    "Students should have some familiarity with Python programming, including at leaset basic exposure to object-oriented programming."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.5 Domain Problem\n",
    "\n",
    "Social media has become a useful source of information about trends in perceptions and attitudes towards various health questions. This module challenges students to learn how to retrieve social media data and to use Natural Language Processing to extract key trends and to classify messages based on those classifications."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.6 Dataset for the case study\n",
    "\n",
    "Tweets about smoking and vaping collected from the Twitter public API during the fall of 2017."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}