part1/03_SpaCy.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "YuZSyQVvl_d-"
   },
   "source": [
    "# SpaCy: NLP Pipeline\n",
    "\n",
    "![](images/spacy_nlp_pipeline.svg)\n",
    "\n",
    "\n",
    "**What we will do**\n",
    "Reference: https://spacy.io/\n",
    "\n",
    "* Tokenization\n",
    "* POS Tagging\n",
    "* NER\n",
    "* Entity linking\n",
    "\n",
    "![](images/spacy-training.svg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tokenization"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "executionInfo": {
     "elapsed": 9621,
     "status": "ok",
     "timestamp": 1603863804546,
     "user": {
      "displayName": "Yashesh Shroff",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggu3bTBcysUxxtmnLI9n_OEGc92nfpZXGRAhDZ5vQ=s64",
      "userId": "12984101785953050215"
     },
     "user_tz": 420
    },
    "id": "i48CEEw6l_eA",
    "outputId": "f0831a64-2739-4915-e498-17b53dfc5531"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[00mUsing Google CoLab = \u001b[93mFalse\n"
     ]
    }
   ],
   "source": [
    "try:\n",
    "    %tensorflow_version 2.x\n",
    "    is_colab = True\n",
    "    !pip install spacy\n",
    "    !python -m spacy download en_core_web_md\n",
    "    !pip install spacy-transformers\n",
    "except:\n",
    "    is_colab = False\n",
    "\n",
    "print(f'\\033[00mUsing Google CoLab = \\033[93m{is_colab}')\n",
    "if (is_colab): print(\"Dependencies installed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vAx9PeMQl_eK"
   },
   "source": [
    "# Spacy: Getting started\n",
    "\n",
    "As discussed in the lecture portion, Python has two main libraries to help with NLP tasks: \n",
    "\n",
    "* [NLTK](https://www.nltk.org/)\n",
    "* [Spacy](https://spacy.io/)\n",
    "\n",
    "SpaCy launched in 2015 and has rapidly become an industry standard, and is a focus of our training. SpaCy provides an industrial grade project that is both open-source and contains community driven integrations (see SpaCy Universe).\n",
    "\n",
    "SpaCy requires you to download language resources (such as models). For the english language, you can use `python -m spacy download en_core_web_sm`. The suffix `_sm` indicates \"small\" model, while `_md` and `_lg` indicate medium and large, respectively and provide more advanced features (we won't need in this tutorial).\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 674
    },
    "executionInfo": {
     "elapsed": 8233,
     "status": "ok",
     "timestamp": 1603861712457,
     "user": {
      "displayName": "Yashesh Shroff",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggu3bTBcysUxxtmnLI9n_OEGc92nfpZXGRAhDZ5vQ=s64",
      "userId": "12984101785953050215"
     },
     "user_tz": 420
    },
    "id": "NsNmfdbWmkxZ",
    "outputId": "7ad799c2-7888-4f57-bcbc-c6d712dfd8d7",
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collecting en-core-web-sm==3.1.0\n",
      "  Downloading https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.1.0/en_core_web_sm-3.1.0-py3-none-any.whl (13.6 MB)\n",
      "\u001b[K     |████████████████████████████████| 13.6 MB 1.3 MB/s eta 0:00:01\n",
      "\u001b[?25hRequirement already satisfied: spacy<3.2.0,>=3.1.0 in /home/yashroff/miniconda3/envs/pytorch/lib/python3.8/site-packages (from en-core-web-sm==3.1.0) (3.1.2)\n",
      "Requirement already satisfied: preshed<3.1.0,>=3.0.2 in /home/yashroff/miniconda3/envs/pytorch/lib/python3.8/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (3.0.5)\n",
      "Requirement already satisfied: spacy-legacy<3.1.0,>=3.0.7 in /home/yashroff/miniconda3/envs/pytorch/lib/python3.8/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (3.0.8)\n",
      "Requirement already satisfied: blis<0.8.0,>=0.4.0 in /home/yashroff/miniconda3/envs/pytorch/lib/python3.8/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (0.7.4)\n",
      "Requirement already satisfied: packaging>=20.0 in /home/yashroff/.local/lib/python3.8/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (20.9)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.38.0 in /home/yashroff/miniconda3/envs/pytorch/lib/python3.8/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (4.62.1)\n",
      "Requirement already satisfied: thinc<8.1.0,>=8.0.8 in /home/yashroff/miniconda3/envs/pytorch/lib/python3.8/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (8.0.10)\n",
      "Requirement already satisfied: srsly<3.0.0,>=2.4.1 in /home/yashroff/miniconda3/envs/pytorch/lib/python3.8/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (2.4.1)\n",
      "Requirement already satisfied: catalogue<2.1.0,>=2.0.4 in /home/yashroff/miniconda3/envs/pytorch/lib/python3.8/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (2.0.6)\n",
      "Requirement already satisfied: wasabi<1.1.0,>=0.8.1 in /home/yashroff/miniconda3/envs/pytorch/lib/python3.8/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (0.8.2)\n",
      "Requirement already satisfied: setuptools in /home/yashroff/miniconda3/envs/pytorch/lib/python3.8/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (52.0.0.post20210125)\n",
      "Requirement already satisfied: pathy>=0.3.5 in /home/yashroff/miniconda3/envs/pytorch/lib/python3.8/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (0.6.0)\n",
      "Requirement already satisfied: typer<0.4.0,>=0.3.0 in /home/yashroff/miniconda3/envs/pytorch/lib/python3.8/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (0.3.2)\n",
      "Requirement already satisfied: requests<3.0.0,>=2.13.0 in /home/yashroff/miniconda3/envs/pytorch/lib/python3.8/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (2.26.0)\n",
      "Requirement already satisfied: cymem<2.1.0,>=2.0.2 in /home/yashroff/miniconda3/envs/pytorch/lib/python3.8/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (2.0.5)\n",
      "Requirement already satisfied: numpy>=1.15.0 in /home/yashroff/.local/lib/python3.8/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (1.19.5)\n",
      "Requirement already satisfied: murmurhash<1.1.0,>=0.28.0 in /home/yashroff/miniconda3/envs/pytorch/lib/python3.8/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (1.0.5)\n",
      "Requirement already satisfied: pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4 in /home/yashroff/miniconda3/envs/pytorch/lib/python3.8/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (1.8.2)\n",
      "Requirement already satisfied: jinja2 in /home/yashroff/miniconda3/envs/pytorch/lib/python3.8/site-packages (from spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (3.0.1)\n",
      "Requirement already satisfied: pyparsing>=2.0.2 in /home/yashroff/.local/lib/python3.8/site-packages (from packaging>=20.0->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (2.4.7)\n",
      "Requirement already satisfied: smart-open<6.0.0,>=5.0.0 in /home/yashroff/miniconda3/envs/pytorch/lib/python3.8/site-packages (from pathy>=0.3.5->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (5.1.0)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/yashroff/.local/lib/python3.8/site-packages (from pydantic!=1.8,!=1.8.1,<1.9.0,>=1.7.4->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (3.7.4.3)\n",
      "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/yashroff/miniconda3/envs/pytorch/lib/python3.8/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (1.26.6)\n",
      "Requirement already satisfied: charset-normalizer~=2.0.0 in /home/yashroff/miniconda3/envs/pytorch/lib/python3.8/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (2.0.4)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /home/yashroff/miniconda3/envs/pytorch/lib/python3.8/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (3.2)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/yashroff/miniconda3/envs/pytorch/lib/python3.8/site-packages (from requests<3.0.0,>=2.13.0->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (2021.5.30)\n",
      "Requirement already satisfied: click<7.2.0,>=7.1.1 in /home/yashroff/miniconda3/envs/pytorch/lib/python3.8/site-packages (from typer<0.4.0,>=0.3.0->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (7.1.2)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /home/yashroff/miniconda3/envs/pytorch/lib/python3.8/site-packages (from jinja2->spacy<3.2.0,>=3.1.0->en-core-web-sm==3.1.0) (2.0.1)\n",
      "\u001b[38;5;2m✔ Download and installation successful\u001b[0m\n",
      "You can now load the package via spacy.load('en_core_web_sm')\n"
     ]
    }
   ],
   "source": [
    "!python -m spacy download en_core_web_sm"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 270
    },
    "executionInfo": {
     "elapsed": 3421,
     "status": "ok",
     "timestamp": 1603862624643,
     "user": {
      "displayName": "Yashesh Shroff",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggu3bTBcysUxxtmnLI9n_OEGc92nfpZXGRAhDZ5vQ=s64",
      "userId": "12984101785953050215"
     },
     "user_tz": 420
    },
    "id": "Nv3udxHCnV9i",
    "outputId": "232eb764-3da7-47cd-ad3c-35132e013bff"
   },
   "outputs": [],
   "source": [
    "!pip install urllib3==1.25.10"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "-SYMMX6bnYtY"
   },
   "outputs": [],
   "source": [
    "!pip show urllib3 | grep version"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "id": "dDn901chl_eL"
   },
   "outputs": [],
   "source": [
    "import spacy\n",
    "nlp = spacy.load('en_core_web_sm')\n",
    "\n",
    "# Use if needed:\n",
    "#spacy.util.get_data_path()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cQjRh0yil_eS"
   },
   "source": [
    "# Tokenization\n",
    "\n",
    "For each word in that sentence _spaCy_ generates a [token](https://spacy.io/api/token) for each word in the sentence. The token fields show the raw text, the root of the word (lemma), the Part of Speech (POS), whether or not its a stop word, and many other things. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 108
    },
    "executionInfo": {
     "elapsed": 487,
     "status": "ok",
     "timestamp": 1603863732252,
     "user": {
      "displayName": "Yashesh Shroff",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggu3bTBcysUxxtmnLI9n_OEGc92nfpZXGRAhDZ5vQ=s64",
      "userId": "12984101785953050215"
     },
     "user_tz": 420
    },
    "id": "ZbUht2qXl_eU",
    "outputId": "cfddc2d6-35a3-4622-c46b-835c0f3b6eb7"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "this this DET DT nsubj xxxx True True\n",
      "is be AUX VBZ ROOT xx True True\n",
      "a a DET DT det x True True\n",
      "beautiful beautiful ADJ JJ amod xxxx True False\n",
      "day day NOUN NN attr xxx True False\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "text = \"this is a beautiful day\"\n",
    "doc = nlp(text)\n",
    "for token in doc:\n",
    "    print(token.text, token.lemma_, token.pos_, token.tag_, token.dep_,\n",
    "          token.shape_, token.is_alpha, token.is_stop)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['gimme', 'that']\n",
      "['gim', 'me', 'that']\n"
     ]
    }
   ],
   "source": [
    "from spacy.symbols import ORTH\n",
    "\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "text = \"gimme that\"\n",
    "doc = nlp(text)  # phrase to tokenize\n",
    "print([w.text for w in doc])  # ['gimme', 'that']\n",
    "\n",
    "# Add special case rule\n",
    "special_case = [{ORTH: \"gim\"}, {ORTH: \"me\"}]\n",
    "nlp.tokenizer.add_special_case(\"gimme\", special_case)\n",
    "\n",
    "# Check new tokenization\n",
    "print([w.text for w in nlp(\"gimme that\")])  # ['gim', 'me', 'that']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gim \t SPECIAL-1\n",
      "me \t SPECIAL-2\n",
      "that \t TOKEN\n"
     ]
    }
   ],
   "source": [
    "tok_exp = nlp.tokenizer.explain(text)\n",
    "for t in tok_exp:\n",
    "    print(t[1], '\\t', t[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[False, False, False, False, False, False, False, False, False]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "[t.is_space for t in nlp('''\"a gimme give me let's \"''')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6Y5IJx92l_eZ"
   },
   "source": [
    "# Numeric representation\n",
    "\n",
    "Let's print the last token and see its _numeric_ representation:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 399
    },
    "executionInfo": {
     "elapsed": 448,
     "status": "ok",
     "timestamp": 1603863734944,
     "user": {
      "displayName": "Yashesh Shroff",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggu3bTBcysUxxtmnLI9n_OEGc92nfpZXGRAhDZ5vQ=s64",
      "userId": "12984101785953050215"
     },
     "user_tz": 420
    },
    "id": "zgFHH4fpl_ea",
    "outputId": "bc86efec-4305-43ee-acad-64baf421b3fe"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The token is from the raw text: \u001b[92mday\u001b[0m\n",
      "Numeric representation:\n",
      "\n",
      "[ 1.0326469  -0.5188544   0.8102132  -0.22484887  0.1622698   0.12331456\n",
      " -0.29505694  0.30728796 -0.5055412  -0.53730524  0.5390411  -0.06072912\n",
      "  0.21190232 -0.570506   -0.7399631   0.76866597  0.62896025  0.14521122\n",
      " -0.7163496   0.0914988  -0.44675386 -0.42669684 -0.30483526 -0.02167813\n",
      " -0.89213675 -0.54698586  0.71880984  0.63821524  0.32498112  0.02454378\n",
      " -0.5876424  -0.642883    0.03008172 -0.3074876   1.6153848   0.1442051\n",
      " -0.07353693 -0.32064044  0.38455236  0.5503072  -0.51853573  0.20107445\n",
      " -1.0938597   0.912225   -0.16106965 -0.4424458  -0.49976707  1.6711009\n",
      "  0.28246796  0.03308608  0.3438756   0.04971349  0.5434291   0.32878944\n",
      " -0.1839002  -0.5033451   0.20727718 -0.37684458 -0.13446403  0.17247891\n",
      "  0.6887866  -0.17390436 -0.7523935  -0.37924802 -0.894167   -0.8755444\n",
      " -0.40961346 -0.4575967   0.40342683  0.50362885  0.31731796 -0.08280959\n",
      "  0.12015444 -0.3551879   1.0481503  -0.23068361 -0.9625367   0.58524495\n",
      " -0.10468513  0.904711   -0.07363549 -0.2497644  -0.1647499   0.41395915\n",
      " -0.7800237  -0.57704836  0.00758557 -0.93664217 -0.14248914 -0.12322472\n",
      " -0.37174043  1.2086246  -0.07419129  0.26703867  0.28164724  0.16014299]\n",
      "\n",
      "The length of the vector is (96,)\n"
     ]
    }
   ],
   "source": [
    "print(f'The token is from the raw text: \\033[92m{token.text}\\033[0m\\nNumeric representation:\\n')\n",
    "print(token.vector)\n",
    "print(f'\\nThe length of the vector is {token.vector.shape}') # 96 length vector"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part-of-speech tagging\n",
    "\n",
    "Requires a model for parsing and tagging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "       Text     Lemma    POS  TAG    DEP  Shape Alpha   Stop\n",
      "0  Doordash  Doordash  PROPN  NNP  nsubj  Xxxxx  True  False\n",
      "       Text     Lemma    POS  TAG    DEP  Shape Alpha   Stop\n",
      "0  Doordash  Doordash  PROPN  NNP  nsubj  Xxxxx  True  False\n",
      "1       and       and  CCONJ   CC     cc    xxx  True   True\n",
      "       Text     Lemma    POS  TAG    DEP   Shape Alpha   Stop\n",
      "0  Doordash  Doordash  PROPN  NNP  nsubj   Xxxxx  True  False\n",
      "1       and       and  CCONJ   CC     cc     xxx  True   True\n",
      "2    AirBnB    AirBnB  PROPN  NNP   conj  XxxXxX  True  False\n",
      "       Text     Lemma    POS  TAG    DEP   Shape Alpha   Stop\n",
      "0  Doordash  Doordash  PROPN  NNP  nsubj   Xxxxx  True  False\n",
      "1       and       and  CCONJ   CC     cc     xxx  True   True\n",
      "2    AirBnB    AirBnB  PROPN  NNP   conj  XxxXxX  True  False\n",
      "3      have      have   VERB  VBP   ROOT    xxxx  True   True\n",
      "       Text     Lemma    POS  TAG    DEP   Shape  Alpha   Stop\n",
      "0  Doordash  Doordash  PROPN  NNP  nsubj   Xxxxx   True  False\n",
      "1       and       and  CCONJ   CC     cc     xxx   True   True\n",
      "2    AirBnB    AirBnB  PROPN  NNP   conj  XxxXxX   True  False\n",
      "3      have      have   VERB  VBP   ROOT    xxxx   True   True\n",
      "4     IPO'd     IPO'd   NOUN  NNS   dobj   XXX'x  False  False\n",
      "       Text     Lemma    POS  TAG    DEP   Shape  Alpha   Stop\n",
      "0  Doordash  Doordash  PROPN  NNP  nsubj   Xxxxx   True  False\n",
      "1       and       and  CCONJ   CC     cc     xxx   True   True\n",
      "2    AirBnB    AirBnB  PROPN  NNP   conj  XxxXxX   True  False\n",
      "3      have      have   VERB  VBP   ROOT    xxxx   True   True\n",
      "4     IPO'd     IPO'd   NOUN  NNS   dobj   XXX'x  False  False\n",
      "5      this      this    DET   DT    det    xxxx   True   True\n",
      "       Text     Lemma    POS  TAG       DEP   Shape  Alpha   Stop\n",
      "0  Doordash  Doordash  PROPN  NNP     nsubj   Xxxxx   True  False\n",
      "1       and       and  CCONJ   CC        cc     xxx   True   True\n",
      "2    AirBnB    AirBnB  PROPN  NNP      conj  XxxXxX   True  False\n",
      "3      have      have   VERB  VBP      ROOT    xxxx   True   True\n",
      "4     IPO'd     IPO'd   NOUN  NNS      dobj   XXX'x  False  False\n",
      "5      this      this    DET   DT       det    xxxx   True   True\n",
      "6      week      week   NOUN   NN  npadvmod    xxxx   True  False\n"
     ]
    }
   ],
   "source": [
    "# Another example\n",
    "import pandas as pd\n",
    "doc2 = nlp(\"Doordash and AirBnB have IPO'd this week\")\n",
    "my_columns = ['Text', 'Lemma', 'POS', 'TAG','DEP', 'Shape', 'Alpha', 'Stop']\n",
    "df = pd.DataFrame(columns = my_columns)\n",
    "\n",
    "for token in doc2:\n",
    "    df = df.append(\n",
    "        pd.Series(\n",
    "        [\n",
    "            token.text,\n",
    "            token.lemma_,\n",
    "            token.pos_,\n",
    "            token.tag_,\n",
    "            token.dep_,\n",
    "            token.shape_,\n",
    "            token.is_alpha,\n",
    "            token.is_stop,            \n",
    "        ],\n",
    "        index = my_columns),\n",
    "        ignore_index = True\n",
    "    )\n",
    "    print(df)\n",
    "#    print(token.text, token.lemma_, token.pos_, token.tag_, token.dep_,\n",
    "#            token.shape_, token.is_alpha, token.is_stop)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tNIc9BOml_ed"
   },
   "source": [
    "# Display\n",
    "\n",
    "Note: Run the following as `display.serve` outside of Jupyter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from spacy import displacy\n",
    "\n",
    "displacy.render(doc, style=\"dep\", jupyter=True)\n",
    "displacy.render(doc, style=\"ent\")\n",
    "# day is shown as a recognized \"DATE\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "RWi7haAJl_ej"
   },
   "source": [
    "### Exercise:\n",
    "\n",
    "Explore different parts of speech & sentence structures. \n",
    "* Show PERSON \n",
    "* Show location\n",
    "\n",
    "Some examples:\n",
    "* \"They met at a cafe in London last year\"\n",
    "* \"Peter went to see his uncle in Brooklyn\"\n",
    "* \"The chicken crossed the road because it was hungry\"\n",
    "* \"The chicken crossed the road because it was narrow\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ZZsB9LKBl_ek"
   },
   "source": [
    "# Similarity of two sentences\n",
    "\n",
    "Let's do the same as above, but mix with two similar sentences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "id": "qwismScsl_em"
   },
   "outputs": [],
   "source": [
    "sentence_list = [\"this is a beautiful day\", \"today is bright and sunny\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "id": "9RCD3AlPl_es"
   },
   "outputs": [],
   "source": [
    "doc_list = list(map(nlp, sentence_list))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 435
    },
    "executionInfo": {
     "elapsed": 559,
     "status": "ok",
     "timestamp": 1603863740690,
     "user": {
      "displayName": "Yashesh Shroff",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggu3bTBcysUxxtmnLI9n_OEGc92nfpZXGRAhDZ5vQ=s64",
      "userId": "12984101785953050215"
     },
     "user_tz": 420
    },
    "id": "3_dh1_Xil_ey",
    "outputId": "135125c0-9ad9-406b-d09d-d7e311f09f50"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\u001b[92mPrinting tokens for \u001b[91m\"this is a beautiful day\"\u001b[0m\n",
      "    text       lemma      pos    tag    dep    shape    is_alpha    is_stop\n",
      "--  ---------  ---------  -----  -----  -----  -------  ----------  ---------\n",
      " 0  this       this       DET    DT     nsubj  xxxx     True        True\n",
      " 1  is         be         AUX    VBZ    ROOT   xx       True        True\n",
      " 2  a          a          DET    DT     det    x        True        True\n",
      " 3  beautiful  beautiful  ADJ    JJ     amod   xxxx     True        False\n",
      " 4  day        day        NOUN   NN     attr   xxx      True        False\n",
      "\n",
      "\u001b[92mPrinting tokens for \u001b[91m\"today is bright and sunny\"\u001b[0m\n",
      "    text       lemma      pos    tag    dep       shape    is_alpha    is_stop\n",
      "--  ---------  ---------  -----  -----  --------  -------  ----------  ---------\n",
      " 0  this       this       DET    DT     nsubj     xxxx     True        True\n",
      " 1  is         be         AUX    VBZ    ROOT      xx       True        True\n",
      " 2  a          a          DET    DT     det       x        True        True\n",
      " 3  beautiful  beautiful  ADJ    JJ     amod      xxxx     True        False\n",
      " 4  day        day        NOUN   NN     attr      xxx      True        False\n",
      " 5  today      today      NOUN   NN     npadvmod  xxxx     True        False\n",
      " 6  is         be         AUX    VBZ    ROOT      xx       True        True\n",
      " 7  bright     bright     ADJ    JJ     acomp     xxxx     True        False\n",
      " 8  and        and        CCONJ  CC     cc        xxx      True        True\n",
      " 9  sunny      sunny      ADJ    JJ     conj      xxxx     True        False\n"
     ]
    }
   ],
   "source": [
    "## Python program to understand the usage of tabulate function for printing tables in a tabular format\n",
    "from tabulate import tabulate\n",
    "import pandas as pd\n",
    "\n",
    "column_names = ['text', 'lemma', 'pos', 'tag', 'dep', 'shape', 'is_alpha', 'is_stop']\n",
    "df = pd.DataFrame(columns = column_names)\n",
    "for doc in doc_list:\n",
    "    print(f'\\n\\033[92mPrinting tokens for \\033[91m\"{doc}\"\\033[0m')\n",
    "    for token in doc:\n",
    "        token_list = [token.text, token.lemma_, token.pos_, token.tag_, token.dep_,\n",
    "                      token.shape_, token.is_alpha, token.is_stop]\n",
    "        token_series = pd.Series(token_list, index = df.columns)\n",
    "        df = df.append(token_series, ignore_index=True)\n",
    "    print(tabulate(df, headers=column_names))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GXlEceRll_e4"
   },
   "source": [
    "# Showing similarity between two sentences\n",
    "\n",
    "1. \"this is a beautiful day\"\n",
    "2. \"this day is bright and sunny\"\n",
    "\n",
    "Note: If you have loaded the small (sm) dataset, you will get the following warning:\n",
    "> UserWarning: [W007] The model you're using has no word vectors loaded, so the result of the Token.similarity method will be based on the tagger, parser and NER, which may not give useful similarity judgements. This may happen if you're using one of the small models, e.g. `en_core_web_sm`, which don't ship with word vectors and only use context-sensitive tensors. You can always add your own word vectors, or use one of the larger models instead if available.\n",
    "\n",
    "Try: \n",
    "* `python -m spacy download en_core_web_md`\n",
    "* or: `python -m spacy download en_core_web_lg`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "xbS1Jh-9l_e5"
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "\n",
    "# choose action = 'ignore' to ignore the small dataset warning\n",
    "warnings.filterwarnings(action = \"ignore\") # \"default\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "executionInfo": {
     "elapsed": 536,
     "status": "ok",
     "timestamp": 1603863743950,
     "user": {
      "displayName": "Yashesh Shroff",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggu3bTBcysUxxtmnLI9n_OEGc92nfpZXGRAhDZ5vQ=s64",
      "userId": "12984101785953050215"
     },
     "user_tz": 420
    },
    "id": "couIc9KHl_e-",
    "outputId": "1c805378-05ec-4b2c-9f49-35bd3737f7ef"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.31264528234238464"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc_list[0].similarity(doc_list[1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 171
    },
    "executionInfo": {
     "elapsed": 706,
     "status": "error",
     "timestamp": 1603913356218,
     "user": {
      "displayName": "Yashesh Shroff",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggu3bTBcysUxxtmnLI9n_OEGc92nfpZXGRAhDZ5vQ=s64",
      "userId": "12984101785953050215"
     },
     "user_tz": 420
    },
    "id": "LUThTsETl_fC",
    "outputId": "051a4229-6ac7-4300-b567-8a987c014d31"
   },
   "outputs": [],
   "source": [
    "nlp_md = spacy.load(\"en_core_web_md\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 207
    },
    "executionInfo": {
     "elapsed": 539,
     "status": "error",
     "timestamp": 1603863745872,
     "user": {
      "displayName": "Yashesh Shroff",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Ggu3bTBcysUxxtmnLI9n_OEGc92nfpZXGRAhDZ5vQ=s64",
      "userId": "12984101785953050215"
     },
     "user_tz": 420
    },
    "id": "6ToOp6jFl_fG",
    "outputId": "178f54cf-a220-4124-c0ed-2ce0a9b53aeb"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8196604942680805"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# try again\n",
    "doc_md_list = list(map(nlp_md, sentence_list))\n",
    "doc_md_list[0].similarity(doc_md_list[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "t5fzgShQl_fL"
   },
   "source": [
    "# Paragraph\n",
    "\n",
    "How do you deal with multiple sentences?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "id": "wqmoLSqNl_fM",
    "outputId": "c81ecfd4-d840-4950-d4ac-f8f58b83bf36"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "> When we went out for ice-cream last summer, the place was \n",
      "packed.\n",
      "> This year, however, things are eerily different.\n",
      "> You can see that \n",
      "the stores are nearly desserted and roads empty like never before.\n",
      "> It's a \n",
      "reality that we are all getting used to, albeit very slowly and reluctantly.\n",
      "> \n",
      "\n"
     ]
    }
   ],
   "source": [
    "text = \"\"\"When we went out for ice-cream last summer, the place was \n",
    "packed. This year, however, things are eerily different. You can see that \n",
    "the stores are nearly desserted and roads empty like never before. It's a \n",
    "reality that we are all getting used to, albeit very slowly and reluctantly.\n",
    "\"\"\"\n",
    "\n",
    "doc = nlp(text)\n",
    "\n",
    "for sent in doc.sents:\n",
    "    print(\">\", sent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5Hj4_fC4l_fa"
   },
   "source": [
    "# Scattertext\n",
    "\n",
    "The following is nice demonstration of the power of SpaCy with text from the Democratic and Republican conventions over the years. This demo is created by \n",
    "derwen.ai using the `scattertext` library. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, install scattertext\n",
    "!pip install scattertext"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "id": "HeNkDM0YzO6s"
   },
   "outputs": [],
   "source": [
    "?nlp.create_pipe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "id": "gmOu3OkRl_ff"
   },
   "outputs": [],
   "source": [
    "import scattertext as st\n",
    "\n",
    "# By default, the nlp English pipeline comes with `tagger`, `parser`, and `NER`\n",
    "if \"merge_entities\" not in nlp.pipe_names:\n",
    "    nlp.add_pipe(\"merge_entities\")\n",
    "\n",
    "if \"merge_noun_chunks\" not in nlp.pipe_names:\n",
    "    nlp.add_pipe(\"merge_noun_chunks\") # nlp.create_pipe(\"merge_noun_chunks\") for spacy.__version__ prior to 3.1\n",
    "\n",
    "convention_df = st.SampleCorpora.ConventionData2012.get_data() \n",
    "corpus = st.CorpusFromPandas(convention_df,\n",
    "                             category_col=\"party\",\n",
    "                             text_col=\"text\",\n",
    "                             nlp=nlp).build()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "eO0PpBGYl_fl"
   },
   "source": [
    "Generate interactive visualization once the corpus is ready:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "id": "rAW5yL2Gl_fm"
   },
   "outputs": [],
   "source": [
    "html = st.produce_scattertext_explorer(\n",
    "    corpus,\n",
    "    category=\"democrat\",\n",
    "    category_name=\"Democratic\",\n",
    "    not_category_name=\"Republican\",\n",
    "    width_in_pixels=1000,\n",
    "    metadata=convention_df[\"speaker\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wvF2hn3ml_ft"
   },
   "source": [
    "Render the visualization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "id": "_ZLqCy9tl_fu",
    "outputId": "6f08dfdb-2a54-4ab7-b9c2-5d8fa0713f15"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "False\n"
     ]
    }
   ],
   "source": [
    "from IPython.display import IFrame\n",
    "from IPython.core.display import display, HTML\n",
    "import sys\n",
    "\n",
    "IN_COLAB = \"google.colab\" in sys.modules\n",
    "print(IN_COLAB)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MM3jJ0pkl_f0"
   },
   "source": [
    "**Use in Google Colab**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "id": "s83lFE99l_f1"
   },
   "outputs": [],
   "source": [
    "if IN_COLAB:\n",
    "    display(HTML(\"<style>.container { width:98% !important; }</style>\"))\n",
    "    display(HTML(html))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QsLosgmLl_f5"
   },
   "source": [
    "**Use in Jupyter**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "file_name = \"foo.html\"\n",
    "\n",
    "with open(file_name, \"wb\") as f:\n",
    "    f.write(html.encode(\"utf-8\"))\n",
    "\n",
    "IFrame(src=file_name, width = 1200, height=700)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "w6AIHKvNvwUT"
   },
   "source": [
    "# The SpaCy universe\n",
    "\n",
    "That's the end of our intro to SpaCy journey. However, as discussed, SpaCy is an open, collaborative project that has a universe of plugins and datasets that make working with it very helpful for a number of use cases. The following is a sampling of the [SpaCy Universe](https://spacy.io/universe):\n",
    " - [Legal: Blackstone](https://spacy.io/universe/project/blackstone)\n",
    " - [Biomedical: Kindred](https://spacy.io/universe/project/kindred)\n",
    " - [Geographic: mordecai](https://spacy.io/universe/project/mordecai)\n",
    " - [Label: Prodigy](https://spacy.io/universe/project/prodigy)\n",
    " - [Edge: spacy-raspberry](https://spacy.io/universe/project/spacy-raspberry)\n",
    " - [Voice: Rasa NLU](https://spacy.io/universe/project/rasa) \n",
    "  - [Transformers: spacy-transformers](https://explosion.ai/blog/spacy-pytorch-transformers) \n",
    "  - [Conference: spaCy IRL 2019](https://irl.spacy.io/2019/)\n",
    "\n",
    "  _Credit: Derwen.ai_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "colab": {
   "name": "02_SpaCy.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}