text_mining.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Text mining\n",
    "\n",
    "To mine text you can use i.e. :\n",
    "* TF-IDF\n",
    "* [TextRank](https://github.com/davidadamojr/TextRank)\n",
    "* [RAKE](https://github.com/aneesha/RAKE)\n",
    "\n",
    "Below I will be using RAKE (Rapid Automtic Keyword Exraction) algorithm. In addition we will use [NTLK](http://www.nltk.org/).\n",
    "\n",
    "In this file we will analyse the text form [this](https://thenextweb.com/artificial-intelligence/2018/01/11/ai-learns-how-to-fool-text-to-speech-thats-bad-news-for-voice-assistants/) article. \n",
    "\n",
    "## Before we start\n",
    "Let's import needed packages and paste some text. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'rake_nltk'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-2-b8097186300c>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0;32mfrom\u001b[0m \u001b[0mrake_nltk\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mRake\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      2\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0mcollections\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mOrderedDict\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m \u001b[0;32mfrom\u001b[0m \u001b[0moperator\u001b[0m \u001b[0;32mimport\u001b[0m \u001b[0mitemgetter\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mModuleNotFoundError\u001b[0m: No module named 'rake_nltk'"
     ]
    }
   ],
   "source": [
    "from rake_nltk import Rake\n",
    "from collections import OrderedDict\n",
    "from operator import itemgetter "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "r = Rake()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How does it work?\n",
    "\n",
    "Rake was made based on observation of reasearchers. They realized that very often keyword contain multiple words, punctation or stop words such as function words *and*, *the* and *of* or other words with low lexical meaning.  \n",
    "\n",
    "### Math behind it\n",
    "First, the text is split into the array of words by the word delimeteres. This array is then split into sequences using phase delimeters and stop word positioning. This is how candidate words are made.\n",
    "\n",
    "#### Keyword scores\n",
    "Next for each of the candidate words we are alco creating the graph of co-occurence\n",
    "\n",
    "![co-occurence_graph.png](https://github.com/konradbjk/Rule-Based-Engine-pyknow/blob/master/graphics/co-occurence_graph.png?raw=true)\n",
    "\n",
    "Next we calculate the frequency of occurance of each of the words. The sore of phase is sum of score of each words in this phase. Score is calculated by dividing the word degree by word's frequency.\n",
    "\n",
    "![img](http://bit.ly/2nAFeJV)\n",
    "\n",
    "So we recive table like this:\n",
    "\n",
    "![score_calculating.png](https://github.com/konradbjk/Rule-Based-Engine-pyknow/blob/master/graphics/score_calculating.png?raw=true)\n",
    "\n",
    "For people who want to know more details I herby suggest reading [this article](https://www.researchgate.net/publication/227988510_Automatic_Keyword_Extraction_from_Individual_Documents)\n",
    "\n",
    "### Keywords extraction\n",
    "\n",
    "First of all let's extract keywords from the given text. Unfortunately this package does not automatically import files but we can code by by our own."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "with open('sample_text.txt', 'r') as f:\n",
    "    text=f.read().replace('\\n', '')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "r.extract_keywords_from_text(text)\n",
    "\n",
    "r.get_ranked_phrases()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But getting the phases aloe is not enought. Let's get also score of the phases. It will tell us more about it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "r.get_ranked_phrases_with_scores()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word degrees\n",
    "\n",
    "Okay we did phases extraction but in many cases this can be not enought. That is why also we can use word degrees (co-occurences of the words. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "word_degrees = r.get_word_degrees()\n",
    "\n",
    "sorted_word_degrees = OrderedDict(sorted(word_degrees.items(), key = itemgetter(1), reverse = True))\n",
    "\n",
    "sorted_word_degrees"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Word occurences\n",
    "\n",
    "As we have word degrees and we also fetched the key phases / words now we would like to get also word occurence in the text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "word_frequency = r.frequency_dist\n",
    "\n",
    "sorted_word_frequency = OrderedDict(sorted(word_frequency.items(), key = itemgetter(1), reverse = True))\n",
    "\n",
    "sorted_word_frequency"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}