LIWC-report-4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Developing the Western Business Sentiment Dictionary - Abstract"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1 Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The mass media enriches our background knowledge, provides us with all kind information, and, definitely, affects our lives. Numerous studies (Klapper [1] is a classic example) have addressed the question of how powerful are the effects of the media on the public.\n",
    "\n",
    "Our research question in particular is: *\"Do the media convey predominantly positive or negative messages about entrepreneurship?\"* In order to answer this inquiry, we analyzed the sentiment associated to business-related sentences extracted from both the New York Times and the Financial Times for the periods January 1999 to December 2014 and January 2003 to December 2014, respectively, that contained certain keywords related to the business world. Upon completing the analysis, our conclusion is *\"Yes, the media convey positive messages.\"* "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 Methodology"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This project was performed in three main phases:\n",
    "\n",
    "1. Human sentiment assessment\n",
    "2. Classifier development\n",
    "3. Sentiment analysis for NYT and FT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Human sentiment assessment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we created a dataset consisted of 8,996 sentences obtained by searching specific terms from The New York Times API and retrieving the related sentences from articles published in 2013 that contained those terms. In order to power an artificial intelligence program able to analyze vast amounts of text and assess the related sentiment, we needed to have human subjects report their reaction to business-related textual excerpts. We had 190 participants of 23 nationalities that assessed the sentiment (very positive, positive, neutral, negative, or very negative) for these sentences by means of the PyBossa crowdsourcing platform.\n",
    "\n",
    "The main goal of this project was to create a reliable sentiment classifier for the business domain. Our working hypothesis was that due to the complexity and rhetoric of the business world, current industry solutions for sentiment analysis would perform poorly. We started by using the renowned LIWC as the de facto standard to evaluate our sentences and compare its results with our pool of participants.\n",
    "\n",
    "LIWC analyzes every sentence and counts word by word. As each target word is processed, the dictionary is searched, looking for a dictionary match with the current target word. If the target word matches the category word, the appropriate word category scale (or scales) for that word is incremented. Due to its hierarchical category arrangement, only the two top categories for emotions (positive emotion and negative emotion) were considered for this experiment.\n",
    "\n",
    "Table 1 shows the results:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## References"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[1] Klapper, J.T. 1960. The Effects of Mass Communication. New York: Free Press."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div>\n",
    "    <table>\n",
    "        <tr><td>Total sentences</td><td>8996</td></tr>\n",
    "        <tr><td>Sentences evaluated with same polarity</td><td>3202</td></tr>\n",
    "        <tr><td>Sentences evaluated with different polarity</td><td>5794</td></tr>\n",
    "        <tr><td>LIWC accuracy</td><td>35.59%</td></tr>\n",
    "        <caption>LIWC accuracy</caption>\n",
    "    </table>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Figure 1 shows the density distribution of sentiments (normalized to the range [-1, +1]) assessed by the participants and by LIWC."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "    <figure>\n",
    "        <img src=\"histogram_b2_den.png\" width=\"600px\"/>\n",
    "        <figcaption>Fig.1 Density distribution of assessed sentiments: participants (blue) vs LIWC (green)</figcaption>\n",
    "    </figure>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The image shows how different are LIWC results from the participant results. LIWC is strongly biased towards 0 (it evaluates a large amount of sentences as neutral). The second remarkable point is that LIWC rarely classifies the sentence as very negative or very possitive, in contrast to the participants.\n",
    "\n",
    "Then, several normality tests were performed - Anderson-Darling, Kolmogorov-Smirnov, and Pearson. Despite the fact that the image above seem to show normal distributions, the tests revealed the opposite. For every test, p-value < 0.05, i. e., neither of the distributions are normal. This is not a surprising result as \"data from the human social world is often not normally distributed.\" (John Canning, \"[Statistics for the Humanities](http://statisticsforhumanities.net/book/)\", 2014).\n",
    "\n",
    "In conclusion, we could not accept LIWC as a sentiment classifier for this project. Although both distributions are slightly biased towards a positive polarity (as the density distribution show), there are much more important differences:\n",
    "1. The density distribution shows that LIWC is strongly biased towards 0 (it evaluates a large amount of sentences as neutral) and it rarely classifies the sentence as very negative or very possitive, in contrast to the participants.\n",
    "3. The normality tests show the participants' distribution is very close to the normal distribution while LIWC's is very far from it.\n",
    "4. Kolmogorov-Smirnov gives a definitive result: participants' and LIWC's results come from different distributions.\n",
    "5. LIWC accuracy (35.59%) is very below the minimum value admissible for any classificator (70%).\n",
    "\n",
    "Therefore, it was necessary to create a more suitable classifier for this case study."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Classifier development"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For this task, we used scikit-learn, a Python library for machine learning that provides multiple classifiers and tools for data mining and data analysis. We did 10-fold cross-validation and trained several classifiers with different parameters. These classifiers were: Ridge, Perceptron, Passive-Aggressive, Stochastic Gradient Descent, Nearest Centroid, Bernoulli Naive Bayes, and Linear Support Vector Classification. Of these, the classifier with best performance was the Nearest Centroid (Euclidean distance and no shrink threshold) with an accuracy of 72%.\n",
    "\n",
    "Firstly, FT provided us with the fulltext articles from 2003 to 2014 in XML. Secondly, we coded two scripts in Python - one to unify the format of all the articles across the different XML files and another one that searched for the articles that contained the business-related keywords. Finally, we analyzed the sentiment of the sentences - the classifier previously trained for the NYT (nearest centroid) assigned a (positive or negative) value to the sentences extracted from the articles.\n",
    "\n",
    "This work was performed by the [CulturePlex Lab](http://www.cultureplex.ca/) with the contributions of Antonio Jiménez-Mavillard, Javier de la Rosa, Adriana Soto-Corominas, and Juan Luis Suárez. The code of this work can be found [here](https://github.com/mavillard/liwc).\n",
    "\n",
    "In order to answer the question *\"Do the media convey predominantly positive or negative messages about entrepreneurship, and if so, does that matter?\"*, we analyzed the sentiment associated to business-related textual excerpts. These excerpts were sentences extracted from the New York Times for the period January 1999 to December 2014 that contained certain keywords related to the business world. Upon completing the analysis, our conclusion is *\"Yes, the media convey positive messages.\"* \n",
    "\n",
    "This part of the project was performed in three main steps:\n",
    "\n",
    "1. Classifier design\n",
    "2. Sentence extraction\n",
    "3. Sentiment analysis\n",
    "\n",
    "Firstly, a nearest centroid classifier was trained with the dataset consisted of 8,996 sentences that our 190 participants had previously assessed. Secondly, we coded a Python script to search for articles, via the NYT API, that contained the business-related keywords. Finally, the classifier assigned a (positive or negative) value to the sentences extracted from the articles."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Overview of the New York Times results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Methodology"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### a) Fulltext access"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unlike the experiment with the NYT, in which we accessed through its API the articles that contained our business-related keywords, we got the fulltext news dataset of the FT from 2003 to 2014 in XML files. These files were provided by the FT's staff, and came from two different sources, the newspaper and ft.com, with different format.\n",
    "\n",
    "The first step was to parse the XML files. For each article, we extracted the following fields:\n",
    "* title\n",
    "* full text\n",
    "* publication date\n",
    "* unique id\n",
    "* source\n",
    "* url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### b) Sentence extraction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we searched every article and collected all the sentences that contained any of the business-related terms. The result was ~half a million sentences organized in a data frame with the following columns:\n",
    "* the sentence itself\n",
    "* the id of the article where the sentence came from\n",
    "* the publication date of the article\n",
    "* the source of the article\n",
    "* the title of the article\n",
    "* the url of the article\n",
    "* the search term\n",
    "* the category of the term (business vocabulary, big company, or new company)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### c) Sentiment analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this phase, we used the nearest centroid classifier obtained in the experiment with the NYT to analyze the sentiment of the whole sentence dataset. The sentiment polarity (positive or negative) was added to the data frame in a new column."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next table shows the total amount of sentences, grouped by term category, and the number of positive and negative sentences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<table>\n",
    "    <caption>General sentiment polarity by term category</caption>\n",
    "    <thead>\n",
    "        <th>category</th><th>negative</th><th>positive</th><td>% neg</td><td>% pos</td><th>total</th>\n",
    "    </thead>\n",
    "    <tbody>\n",
    "        <tr><td>executive/manager</td><td>188155</td><td>221574</td><td>46%</td><td>54%</td><td>409729</td></tr>\n",
    "        <tr><td>entrepreneur/founder</td><td>12703</td><td>35343</td><td>26%</td><td>74%</td><td>48046</td></tr>\n",
    "        <tr><td>big companies</td><td>106342</td><td>156621</td><td>40%</td><td>60%</td><td>262963</td></tr>\n",
    "        <tr><td>new companies</td><td>8780</td><td>19644</td><td>31%</td><td>69%</td><td>28424</td></tr>\n",
    "    </tbody>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Figure 1 shows this result over the period 2003 to 2014."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "    <figure>\n",
    "        <img src=\"ft/executive_manager_total.png\" width=\"600\"/>\n",
    "        <img src=\"ft/entrepreneur_founder_total.png\" width=\"600\"/>\n",
    "        <img src=\"ft/big_companies_total.png\" width=\"600\"/>\n",
    "        <img src=\"ft/new_companies_total.png\" width=\"600\"/>\n",
    "        <figcaption>Fig.1 Distribution of sentences by category over the period 2003 to 2014</figcaption>\n",
    "    </figure>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Figure 2 shows their respective positive rate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "    <figure>\n",
    "        <img src=\"ft/categories_pos_rate.png\" width=\"600\"/>\n",
    "        <figcaption>Fig.2 Positive rate of sentences by category over the period 2003 to 2014</figcaption>\n",
    "    </figure>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Figure 3 shows the total amount of sentences, the number of positive, and negative sentences by profession."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "    <figure>\n",
    "        <img src=\"ft/executive_total.png\" width=\"600\"/>\n",
    "        <img src=\"ft/manager_total.png\" width=\"600\"/>\n",
    "        <img src=\"ft/entrepreneur_total.png\" width=\"600\"/>\n",
    "        <img src=\"ft/founder_total.png\" width=\"600\"/>\n",
    "        <figcaption>Fig.3 Distribution of sentences by profession over the period 2003 to 2014</figcaption>\n",
    "    </figure>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Figure 4 shows their respective positive rate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "    <figure>\n",
    "        <img src=\"ft/professions_pos_rate.png\" width=\"600\"/>\n",
    "        <figcaption>Fig.4 Positive rate of sentences by profession over the period 2003 to 2014</figcaption>\n",
    "    </figure>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Figure 5 shows the total amount of sentences, the number of positive, and negative sentences for some big companies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "    <figure>\n",
    "        <img src=\"ft/chevron_total.png\" width=\"600\"/>\n",
    "        <img src=\"ft/ford_total.png\" width=\"600\"/>\n",
    "        <img src=\"ft/microsoft_total.png\" width=\"600\"/>\n",
    "        <img src=\"ft/mckinsey_total.png\" width=\"600\"/>\n",
    "        <figcaption>Fig.5 Distribution of sentences for big companies over the period 2003 to 2014</figcaption>\n",
    "    </figure>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Figure 6 shows their respective positive rate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "    <figure>\n",
    "        <img src=\"ft/some_big_companies_pos_rate.png\" width=\"600\"/>\n",
    "        <figcaption>Fig.6 Positive rate of sentences for big companies over the period 2003 to 2014</figcaption>\n",
    "    </figure>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Figure 7 shows the total amount of sentences, the number of positive, and negative sentences for some new companies."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "    <figure>\n",
    "        <img src=\"ft/facebook_total.png\" width=\"600\"/>\n",
    "        <img src=\"ft/instagram_total.png\" width=\"600\"/>\n",
    "        <img src=\"ft/linkedin_total.png\" width=\"600\"/>\n",
    "        <img src=\"ft/netflix_total.png\" width=\"600\"/>\n",
    "        <img src=\"ft/spotify_total.png\" width=\"600\"/>\n",
    "        <img src=\"ft/twitter_total.png\" width=\"600\"/>\n",
    "        <figcaption>Fig.7 Distribution of sentences for new companies over the period 2003 to 2014</figcaption>\n",
    "    </figure>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Figure 8 shows their respective positive rate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "    <figure>\n",
    "        <img src=\"ft/some_new_companies_pos_rate.png\" width=\"600\"/>\n",
    "        <figcaption>Fig.8 Positive rate of sentences for new companies over the period 2003 to 2014</figcaption>\n",
    "    </figure>\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Conclusions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The previous figures show the amount of sentences that contain certain business-related keywords extracted from the FT. They also show their sentiment polarity, performed by a nearest centroid classifier with a high accuracy (71%).\n",
    "\n",
    "The image 2 show that the sentiment associated to new companies is more positive than the associated to the old ones. The sentiment associated to professions like entrepreneur or founder are more positive than the associated to executive or manager. This can be seen in more detail in the image 4.\n",
    "\n",
    "The images 6 and 8 reveal that old companies range from low (0.3) to high (0.7) values of positiveness, while new companies remain steady in high levels of positiveness, around 0.6-0.9.\n",
    "\n",
    "Our conclusion is that the media convey a predominantly possitive sentiment about entrepreneurship."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.4.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}