4projects.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Summarization: NLP mini-projects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### <span style=\"color:coral\">Text summarization (with NLTK)</span>\n",
    "\n",
    "Another great library to work with texts is [NLTK](http://www.nltk.org/), which stands for Natural Language Toolkit. We have now extracted a large set of news on a given topic and would like to extract the most informative parts: out of each text we would like to exatract the most informative sentence. This type of summarization of texts is called **extractive**.\n",
    "\n",
    "Take a look at this video: [Hillary Clinton's concession speech](https://www.vox.com/2016/11/9/13570328/hillary-clinton-concession-speech-full-transcript-2016-presidential-election). The webpage already provides the full transcript.\n",
    "\n",
    "\n",
    "\n",
    "We will approach this task as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "corpus length: 6390\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'thank you. thank you all very much. thank you so much.\\n\\nvery rowdy group. thank you, my friends. thank you. thank you. thank you so very much for being here. i love you all, too.\\n\\nlast night i congratulated donald trump and offered to work with him on behalf of our country.\\n\\ni hope that he will be a'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# import text\n",
    "\n",
    "text=open('data/trainhillary.txt').read().lower().replace('\\xa0',' ')\n",
    "print('corpus length:', len(text))\n",
    "text[0:300]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "from collections import defaultdict\n",
    "stopWords = set(stopwords.words(\"english\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['thank you.',\n",
       " 'thank you all very much.',\n",
       " 'thank you so much.',\n",
       " 'very rowdy group.',\n",
       " 'thank you, my friends.',\n",
       " 'thank you.',\n",
       " 'thank you.',\n",
       " 'thank you so very much for being here.',\n",
       " 'i love you all, too.',\n",
       " 'last night i congratulated donald trump and offered to work with him on behalf of our country.']"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentences = nltk.sent_tokenize(text)\n",
    "len(sentences)\n",
    "sentences[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "word_sent = [nltk.word_tokenize(s.lower()) for s in sentences]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['thank', 'you', '.'],\n",
       " ['thank', 'you', 'all', 'very', 'much', '.'],\n",
       " ['thank', 'you', 'so', 'much', '.'],\n",
       " ['very', 'rowdy', 'group', '.'],\n",
       " ['thank', 'you', ',', 'my', 'friends', '.']]"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "word_sent[0:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "353"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#compute frequencies \n",
    "freq = defaultdict(int)\n",
    "for sentence in word_sent:\n",
    "    for word in sentence:\n",
    "        if word not in stopWords:\n",
    "            freq[word] +=1\n",
    "len(freq)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "77.0"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = float(max(freq.values()))\n",
    "m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [],
   "source": [
    "for word in freq.keys():\n",
    "    freq[word] = freq[word]/m"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "min_cut=0.2\n",
    "max_cut=0.8\n",
    "freq_new = defaultdict(int)\n",
    "for word in freq.keys():\n",
    "    if not freq[word] > max_cut or freq[word] < min_cut:\n",
    "        freq_new[word] = freq[word]\n",
    "freq = freq_new\n",
    "del freq_new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "351"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(freq)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "ranking = defaultdict(int)\n",
    "for i, sentence in enumerate(word_sent):\n",
    "        for word in sentence:\n",
    "            if word in freq:\n",
    "                ranking[i] +=freq[word]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "fragment"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[35]\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'we’ve spent a year and a half bringing together millions of people from every corner of our country to say with one voice that we believe that the american dream is big enough for everyone—for people of all races, and religions, for men and women, for immigrants, for lgbt people, and people with disabilities.'"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from heapq import nlargest\n",
    "sentences_index = nlargest(1, ranking, key=ranking.get)\n",
    "print(sentences_index)\n",
    "sentences[sentences_index[0]]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "## All of it in one function\n",
    "from heapq import nlargest\n",
    "import nltk\n",
    "from nltk.corpus import stopwords\n",
    "from collections import defaultdict\n",
    "\n",
    "stopWords = set(stopwords.words(\"english\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": true,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "def summarize_text(text, stopWords, min_cut, max_cut, ntop=1):\n",
    "   \n",
    "    sentences = nltk.sent_tokenize(text)\n",
    "    \n",
    "    word_sent = [nltk.word_tokenize(s.lower()) for s in sentences]\n",
    "    \n",
    "    # compute frequencies \n",
    "    freq = defaultdict(int)\n",
    "    for sentence in word_sent:\n",
    "        for word in sentence:\n",
    "            if word not in stopWords:\n",
    "                freq[word] +=1\n",
    "\n",
    "    # normilize frequencies \n",
    "    m = float(max(freq.values()))\n",
    "    for word in freq.keys():\n",
    "        freq[word] = freq[word]/m\n",
    " \n",
    "    # cut off too frequent or too rare words\n",
    "    freq_new = defaultdict(int)\n",
    "    for word in freq.keys():\n",
    "        if not freq[word] >= max_cut or freq[word] <= min_cut:\n",
    "            freq_new[word] = freq[word]\n",
    "    freq = freq_new\n",
    "    del freq_new\n",
    "    \n",
    "    # rank sentences\n",
    "    ranking = defaultdict(int)\n",
    "    for i, sentence in enumerate(word_sent):\n",
    "        for word in sentence:\n",
    "            if word in freq:\n",
    "                ranking[i] +=freq[word]\n",
    "                \n",
    "    sentences_index = nlargest(ntop, ranking, key=ranking.get)\n",
    "    summary = [sentences[sentences_index[ind]] for ind in range(len(sentences_index))]\n",
    "    return summary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false,
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "SUMMARIZING SENTENCE : \n",
      "we’ve spent a year and a half bringing together millions of people from every corner of our country to say with one voice that we believe that the american dream is big enough for everyone—for people of all races, and religions, for men and women, for immigrants, for lgbt people, and people with disabilities.\n"
     ]
    }
   ],
   "source": [
    "min_cut = 0.2\n",
    "max_cut = 0.8\n",
    "ntop = 1\n",
    "summary = summarize_text(text, stopWords, min_cut, max_cut, ntop)\n",
    "print('SUMMARIZING SENTENCE : \\n' + str(summary[0]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}