practical_excercises.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Practical Introduction to Spacy Pattern Matching"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As this is a practical introduction let's get started with spacy! I have provided a few examples that you can just execute in this jupyter notebook.\n",
    "In the end you will find some further examples and tasks that you could try out yourself.\n",
    "\n",
    "We will use span_ruler by spacy. The pattern definition taks are useful for all other approaches though.\n",
    "\n",
    "We will cover: \n",
    "- Installing all the packages + getting started with spacy\n",
    "- introductory example\n",
    "- Pattern exploration: sandbox\n",
    "- spacy pattern elements\n",
    "- Further examples\n",
    "- Quiz / Example Tasks\n",
    "- Applying patterns to a DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Install all the packages + getting started with spacy\n",
    "To ensure the provided code works for you, you need to install the required python packages in the requirements.txt and install the spacy model we want to use. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Install packages + getting started with spacy\n",
    "# I am using a requirements.txt file to install the packages. It is in the same folder as this file you are reading.\n",
    "%pip install -r requirements.txt\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# alternative way to install the packages:\n",
    "# !pip install spacy\n",
    "# !pip install pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# all the imports we will need\n",
    "import spacy\n",
    "import pandas as pd\n",
    "import json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before one can use the spacy model we need to download it. \n",
    "From the name you can see what you download: the language is english (en), the size small (sm)\n",
    "\n",
    "See more models and pipelines here: https://spacy.io/models"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# before you can use a spacy model you need to download it once\n",
    "!python3 -m spacy download en_core_web_sm\n",
    "\n",
    "# alternative download version\n",
    "# spacy.cli.download('en_core_web_sm')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Links: \n",
    "- Spacy Rule-based matching (https://spacy.io/usage/rule-based-matching)\n",
    "- Spacy Span Ruler (https://spacy.io/usage/rule-based-matching#spanruler)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introductory examples"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = spacy.load(\"en_core_web_sm\") # we load the model\n",
    "# german version: nlp = spacy.load('de_core_news_sm') -> make sure you have it downloaded first\n",
    "\n",
    "\n",
    "config_ = {\"spans_key\": None, \"overwrite\": False } # we can define our configuration\n",
    "\n",
    "ruler = nlp.add_pipe(\"span_ruler\") # we define the ruler by adding the span_ruler to the pipeline #, config = config_\n",
    "\n",
    "# we define the patterns\n",
    "patterns = [\n",
    "         {\n",
    "        \"label\": \"Test concept\",\n",
    "        \"pattern\": [\n",
    "            {\n",
    "                \"LOWER\": \"hello\"\n",
    "            },\n",
    "            \n",
    "            {\n",
    "                \"LOWER\": \"world\"\n",
    "            },\n",
    "        \n",
    "        ]\n",
    "    }]\n",
    "ruler.add_patterns(patterns)\n",
    "\n",
    "text = \"hello world in Germany\"\n",
    "    \n",
    "doc = nlp(text.lower()) # we can also disable named entity recognition with doc = nlp(text.lower(), disable = [\"ner\"])\n",
    "\n",
    "extraction = [(span.label_, span.text) for span in doc.spans[\"ruler\"]]\n",
    "\n",
    "extraction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# -> now: Entity Ruler example\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "ruler = nlp.add_pipe(\"entity_ruler\") # entity ruler this time, lets you add named entities based on pattern dictionaries\n",
    "\n",
    "# we define the patterns\n",
    "patterns = [\n",
    "         {\n",
    "        \"label\": \"Test concept\",\n",
    "        \"pattern\": [\n",
    "            {\n",
    "                \"LOWER\": \"hello\"\n",
    "            },\n",
    "            \n",
    "            {\n",
    "                \"LOWER\": \"germany\"\n",
    "            },\n",
    "        \n",
    "        ]\n",
    "    }]\n",
    "ruler.add_patterns(patterns)\n",
    "\n",
    "text = \"hello world in Germany\"\n",
    "    \n",
    "doc = nlp(text.lower())\n",
    "\n",
    "extraction = [(ent.text, ent.label_) for ent in doc.ents] # -> the match with germany is more accurate and therefore it is chosen to be matched, BUT: we only have one match\n",
    "\n",
    "extraction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can also have a look at entities in the text\n",
    "\n",
    "for ent in doc.ents:\n",
    "    print(ent.text, ent.start_char, ent.end_char, ent.label_) # -> we find a GPE entity (Geopolitical entity) = Germany\n",
    "\n",
    "# Info: more about entity recognition: https://spacy.io/usage/linguistic-features#named-entities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Pattern exploration: sandbox\n",
    "I have provided a 'sandbox' for you. This is what one would call a testing environment where (hopefully) nothing goes wrong. You can explore pattern ideas and test if they find what you expect. Basically a playground for you to try things out.\n",
    "\n",
    "Spacy also provides their version of a sandbox/ Testing tool here: https://demos.explosion.ai/matcher\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# I have written the previous example as function to make testing patterns easier\n",
    "# Just pass the text and the pattern as a list of dictionaries to the function\n",
    "\n",
    "def test_pattern(text: str, pattern: list):\n",
    "    '''\n",
    "    Function to test patterns with spacy\n",
    "    text: str: text to be tested\n",
    "    pattern: list: list of dictionaries with patterns\n",
    "    '''\n",
    "    nlp = spacy.load(\"en_core_web_sm\")\n",
    "    ruler = nlp.add_pipe(\"span_ruler\")\n",
    "    patterns = pattern\n",
    "    ruler.add_patterns(patterns)\n",
    "\n",
    "    doc = nlp(text) # to lower text first: doc = nlp(text.lower())\n",
    "\n",
    "    extraction = [(span.label_, span.text) for span in doc.spans[\"ruler\"]]\n",
    "\n",
    "    return extraction\n",
    "\n",
    "# for example:\n",
    "test_pattern(text = \"test case for the function\", pattern = [{\n",
    "        \"label\": \"Test concept\",\n",
    "        \"pattern\": [\n",
    "            {\n",
    "                \"TEXT\": \"test\"\n",
    "            },\n",
    "            \n",
    "            {\n",
    "                \"TEXT\": \"case\"\n",
    "            },\n",
    "        \n",
    "        ]\n",
    "    }])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Spacy pattern elements\n",
    "\n",
    "Spacy offers available token attributes. So attributes (names) that you can use to define patterns\n",
    "\n",
    "See the list here: https://spacy.io/usage/rule-based-matching#adding-patterns-attributes\n",
    "\n",
    "We will look into a few common ones and later in some more advanced ones.\n",
    "\n",
    "the most basic ones are 'TEXT' and 'LOWER\n",
    "\n",
    "TEXT = The exact verbatim text of a token (which we have used so far)\n",
    "\n",
    "LEMMA = Base form of the token, with no inflectional suffixes. There are different ways to do lemmatization (the process of getting the base word, see an overview for example here: https://www.geeksforgeeks.org/python-lemmatization-with-nltk/)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_pattern(text = \"The quick brown foxes are jumping over the lazy dogs\", pattern = [{\n",
    "        \"label\": \"Testing concept\",\n",
    "        \"pattern\": [\n",
    "            {\n",
    "                \"LEMMA\": \"jump\" # because we use lemma here we can also match jumping as its base form is jump\n",
    "            },\n",
    "            \n",
    "            {\n",
    "                \"LOWER\": \"over\"\n",
    "            },\n",
    "        \n",
    "        ]\n",
    "    }])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can also look at the lemma \n",
    "doc = nlp(\"The quick brown foxes are jumping over the lazy dogs.\")\n",
    "[token.lemma_ for token in doc]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# wild cards -> you do not care what is matched, makes sense if you do not know filler words\n",
    "# Info: https://spacy.io/usage/rule-based-matching#adding-patterns-wildcard\n",
    "\n",
    "test_pattern(text = \"The quick brown foxes are jumping over the lazy dogs\", pattern = [{\n",
    "        \"label\": \"Testing concept\",\n",
    "        \"pattern\": [\n",
    "            {\n",
    "                \"TEXT\": \"quick\"\n",
    "            },\n",
    "            {}, # this is the \"wild card\" -> we do not care if there is for example another \n",
    "            # adjective in between the words\n",
    "            \n",
    "            {\n",
    "                \"LEMMA\": \"fox\"\n",
    "            },\n",
    "        \n",
    "        ]\n",
    "    }])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# IN / NOT_IN\n",
    "# INfo: https://spacy.io/usage/rule-based-matching#adding-patterns-attributes-extended\n",
    "\n",
    "# we for example accept different ways of writing a word\n",
    "# in german a typical case is accepting: 'Ärzte' but also 'Aerzte' -> you can combine them into one pattern instead of two\n",
    "# this is especially useful when you have a lot of patterns that are similar and long\n",
    "\n",
    "test_pattern(text = \"they work at the doctor, they work at the school\", pattern = [{\n",
    "        \"label\": \"Testing concept\",\n",
    "        \"pattern\": [\n",
    "            {\n",
    "                \"TEXT\": \"work\"\n",
    "            },\n",
    "            {\"TEXT\": \"at\"}, \n",
    "            {}, # we do not care for filler words\n",
    "            \n",
    "            {\n",
    "                \"TEXT\": {\"IN\": [\"doctor\", \"school\"]} # we want to catch multiple phrases and we are for example just interested that \n",
    "                # somebody works and not where\n",
    "            },\n",
    "        \n",
    "        ]\n",
    "    }])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can also match dates / commas etc. with patterns\n",
    "\n",
    "test_pattern(text = \"I was born on January 12, 2001\", pattern = [{\n",
    "        \"label\": \"Testing concept\",\n",
    "        \"pattern\": \n",
    "           [{\"IS_ALPHA\": True}, {\"IS_DIGIT\": True}, {\"IS_PUNCT\": True}, {\"IS_DIGIT\": True}]\n",
    "    }])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# another example with a comma\n",
    "test_pattern(text = \"Hello, world! Hello world!\", pattern = [{\n",
    "        \"label\": \"Testing concept\",\n",
    "        \"pattern\": \n",
    "           [{\"LOWER\": \"hello\"}, {\"IS_PUNCT\": True}, {\"LOWER\": \"world\"}]\n",
    "    }])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further examples + quiz"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# operators and quantifiers I: negation\n",
    "\n",
    "# something should not appear - good for catching negations\n",
    "test_pattern(text = \"Are you going to wear makeup today? No, I was opting for no makeup\", pattern = [{\n",
    "        \"label\": \"only positive formulations\",\n",
    "\"pattern\": [\n",
    "            {\n",
    "                \"OP\": \"!\",\n",
    "                \"TEXT\": {  # here you can declare what should not be in the text\n",
    "                    \"IN\": [\n",
    "                        \"not\", \n",
    "                        \"without\",\n",
    "                        \"avoiding\",\n",
    "                        \"no\"\n",
    "                    ]\n",
    "                }\n",
    "            },\n",
    "            {\n",
    "                \"TEXT\": \"makeup\"\n",
    "            },\n",
    "        ]\n",
    "    },\n",
    "        \n",
    "        ])\n",
    "\n",
    "# -> we do not catch the 'no makeup' part"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# operators and quantifiers II: optional pattern\n",
    "\n",
    "test_pattern(text = \"I  like dogs. I like big fluffy dogs, I like small fast dogs. I like slow dogs: they are funny\", pattern = [{\n",
    "        \"label\": \"any dog description\",\n",
    "\"pattern\": [{\"TEXT\": \"like\"},\n",
    "{\n",
    "                \"OP\": \"?\",\n",
    "                \"POS\": \"ADJ\" # we are matching adjectives, but we do not care if there is one or not\n",
    "            },\n",
    "            {\n",
    "                \"OP\": \"?\",\n",
    "                \"POS\": \"ADJ\" # we are matching adjectives, but we do not care if there is one or not\n",
    "            },\n",
    "            {\n",
    "                \"LEMMA\": \"dog\"\n",
    "            },\n",
    "        ]\n",
    "    },\n",
    "        \n",
    "        ])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Other operators and quantifiers include:\n",
    "\n",
    "- require the pattern to match 1 or more times +\n",
    "- allow the pattern to match zero or more times *\n",
    "- require the pattern to match exactly n times {n}\n",
    "... \n",
    "\n",
    "See here: Info: official documentation: https://spacy.io/usage/rule-based-matching#quantifiers"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fuzzy matching\n",
    "# Fuzzy matching allows you to match tokens with alternate spellings, typos, etc. without specifying every possible variant.\n",
    "# Info: https://spacy.io/usage/rule-based-matching#adding-patterns-attributes-extended\n",
    "\n",
    "test_pattern(text = \"emma lives in the uk and her favourite icecream is vanilla. Sam lives in the US and his favorite icecream is chocolate\",\n",
    " pattern = [{\n",
    "        \"label\": \"Testing concept\",\n",
    "        \"pattern\": [\n",
    "            {\"TEXT\": {\"FUZZY\": \"favorite\"}}, # we do not care if favorite is written the american or british way\n",
    "            {\"TEXT\": \"icecream\"}, \n",
    "        \n",
    "        ]\n",
    "    }])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fuzzy matching: being precise\n",
    "# -> to be more precise you can say how many elements can differ \n",
    "test_pattern(text = '''emma lives in the uk and her favourite icecream is vanilla. \n",
    "                    Sam lives in the US and his favorite icecream is chocolate.\n",
    "                    Rebecca made a typo and her favvourite icecream is strawberry''',\n",
    " pattern = [{\n",
    "        \"label\": \"Testing concept\",\n",
    "        \"pattern\": [\n",
    "            {\"TEXT\": {\"FUZZY1\": \"favorite\"}}, # we allow one element to differ\n",
    "            {\"TEXT\": \"icecream\"}, \n",
    "        \n",
    "        ]\n",
    "    }])\n",
    "# -> we do not find Rebeccas icecream choice as it differs by two elements\n",
    "# fuzzy matching is useful for words with a hyphen (-) or multiple ways of writing something"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fuzzy matching: one bit to far! -> be sure to be precise about what you match \n",
    "test_pattern(text = \"we should go to the theatre soon! Yes, this idea is great\",\n",
    " pattern = [{\n",
    "        \"label\": \"Testing concept\",\n",
    "        \"pattern\": [\n",
    "            {\"TEXT\": {\"FUZZY6\": \"theatre\"}}, # we allow one element to differ\n",
    "        \n",
    "        ]\n",
    "    }])\n",
    "# -> a lot more is matched as the fuzzy matching is too broad"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Regex: A regular expression (shortened as regex or regexp), sometimes referred to as rational expression, is a sequence of characters that specifies a match pattern in text. (Definition from Wikipedia). If spacy pattern matching is not enough you can also use regex to define patterns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# matching with regex\n",
    "# Info: https://spacy.io/usage/rule-based-matching#regex\n",
    "\n",
    "# Find words starting with a specific letter\n",
    "test_pattern(text = \"Have you seen my phone? I did not see it. Do you have it in your pocket?\",\n",
    " pattern = [{\n",
    "        \"label\": \"Testing concept\",\n",
    "        \"pattern\": [{\"TEXT\": {\"REGEX\": \"^p\"}}]\n",
    "    }])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Quiz / Example Tasks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1) Find your full name in a text\n",
    "2) try matching this date format: '2021-01-12'\n",
    "3) try finding email addresses\n",
    "4) try finding currency formats, like: '$100', '20€' or '25 Euro'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "other examples that might be useful:\n",
    "- hashtags and emojis: https://spacy.io/usage/rule-based-matching#example3\n",
    "- phone numbers: https://spacy.io/usage/rule-based-matching#example2\n",
    "- regex playground / tester: https://regex101.com/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will switch from applying the pattern matching to sample strings to datasets. This is a more real-life application to what you would do in an analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Applying patterns to a DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we will be using an example dataframe from kaggle about Youtube comments\n",
    "\n",
    "# Source: https://www.kaggle.com/datasets/ahsenwaheed/youtube-comments-spam-dataset\n",
    "\n",
    "# I have created a few patterns with the help of ChatGPT"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the data + look at the first few rows\n",
    "df = pd.read_csv(\"Youtube-Spam-Dataset.csv\")\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the patterns\n",
    "with open(file=\"youtube_spam_patterns.json\", mode=\"r\") as fp:\n",
    "    patterns = json.load(fp=fp)\n",
    "\n",
    "\n",
    "# create the pipeline\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "ruler = nlp.add_pipe(\"span_ruler\")\n",
    "ruler.add_patterns(patterns)\n",
    "\n",
    "\n",
    "extractions = []\n",
    "\n",
    "# apply the patterns to the content column\n",
    "for index, row in df.iterrows():\n",
    "    doc = nlp(row[\"CONTENT\"])\n",
    "    extraction = list(set([span.label_ for span in doc.spans[\"ruler\"]]))\n",
    "    # we are interested in the comment id and the extraction\n",
    "    for entry in extraction:\n",
    "        extractions.append({\"id\": row[\"COMMENT_ID\"], \"extraction\": entry})\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "extractions = pd.DataFrame(extractions)\n",
    "extractions.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# further analysis of the results, like how many times a pattern was matched\n",
    "extractions[\"extraction\"].value_counts()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "base",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}