question-answering-application-with-weaviate-workshop/question-answering-application-with-weaviate-workshop.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Build your first Question Answering Application with Vector Search Engine **Weaviate**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction \n",
    "\n",
    "Welcome to this workshop about Vector Databases and Weaviate!\n",
    "\n",
    "This notebook is a hands-on tutorial to be used alongside an interactive Weaviate workshop. In this workshop and with this notebook you: learn what vector search is, perform your first semantic search with a prepared demo dataset and build your own vector search application with Question Answering. We'll use a dataset of newspaper articles, but you are free to use your own dataset instead! Besides running each cell, there are some **exercises** defined, in which you need to take some action like coming up with data sources, finishing a data schema and making creative queries.\n",
    "\n",
    "You can also [open the file as Colab notebook](https://colab.research.google.com/github/semi-technologies/weaviate-examples/blob/main/question-answering-application-with-weaviate-workshop/question-answering-application-with-weaviate-workshop.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Table of Contents\n",
    "1. What is a Vector Database?\n",
    "    1. Perform your first semantic search\n",
    "2. Build your first vector search application\n",
    "    1. Create a Weaviate cluster and connect to it\n",
    "    2. Get data and analyze it\n",
    "    3. Make a data schema\n",
    "    4. Vectorize & upload data\n",
    "    5. Query data\n",
    "3. Bonus: automatic data classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. What is a Vector Database?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Traditional search engines perform a *keyword-based search*. Such search engines return results that contain an exact match or a close variation of a search query, like searching for papers about *“Covid-19“*. Those *keyword-based* search engines work well in some cases, but fall short in use cases where *semantics* play a role. This means that traditional search engines do not take the actual meaning of the query and documents or data into account. Documents in a database that might be of interest might not be found, because different wording is used. This means that, for example, a paper with the word *“coronavirus”* in it will not be found with the search term *“Covid-19“*.\n",
    "\n",
    "*Vector-based search* (or semantic search, neural search) tries to solve this problem. Vector search engines like [Weaviate](https://weaviate.io/) retrieve documents by meaning, not by exact matching keywords. With machine learning models like BERT or ResNet50, a vector database converts data objects (text, images, etc) to embeddings. Embeddings are vector representations, which are stored in a high-dimensional vector space. The following image shows a simplified example of embedded concepts and images. Note that some of the name *“cat”* lies close to an image of a cat. And that *“dog”* lies closer to *“cat”* than *“banana“*. \n",
    "\n",
    "![Shows a 3 dimensional space with words and images placed according to their meaning](vectors-3d-multi.png \"Vector database representation\")\n",
    "\n",
    "**Similarity search**\n",
    "\n",
    "If we would search for *“kitten”* in our simple database, the machine learning model would calculate a vector for this word, and return the closest data points. Although there is no datapoint matching exactly the word *“kitten“*, the vector search engine would return objects like the image of the cat, because this is *semantically* close to the query. This is visualized in the following figure:\n",
    "\n",
    "![Shows a 3 dimensional space with a search query and words and images placed according to their meaning](vectors-3d-multi-query.png \"Vector search representation\")\n",
    "\n",
    "**Using Weaviate to do a vector-based search**\n",
    "\n",
    "Weaviate is an open-source vector-based search engine. With Weaviate you can perform a semantic search to various data types (text, images, etc) as described above. To embed data and queries, you can use out-of-the-box models, use open models (e.g. from HuggingFace), or connect your own machine learning model. Weaviate is horizontally scalable, which enables you to scale ML models and search through big data very fast.\n",
    "\n",
    "**Try out a semantic search demo**\n",
    "\n",
    "To get a glimpse of how semantic search in practice works, you can try out the following demo. This demo contains a small dataset of around 3500 news articles, which are embedded with a [BERT model](https://huggingface.co/dbmdz/bert-large-cased-finetuned-conll03-english). You can access this dataset on https://demo.dataset.playground.semi.technology. Using the [Weaviate Console](https://console.semi.technology/), you can query this dataset. You can use [this link](https://link.semi.technology/37kMdjR) to access the GraphQL query module on this dataset directly.\n",
    "\n",
    "With the following GraphQL query, you search the dataset for *Articles* that are around the topic of *“kitten”*. The query *“kitten”* will be vectorized using the BERT model, and *Articles* that are close to this vector will be retrieved. The title of the top articles along with the calculated *“certainty”* (ranging from 0.0-1.0) are shown. \n",
    "\n",
    "```graphql\n",
    "{\n",
    "  Get{\n",
    "    Article(\n",
    "      nearText: {\n",
    "        concepts: [\"kitten\"]\n",
    "      }\n",
    "    ){\n",
    "      title\n",
    "      _additional {\n",
    "        certainty\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "```\n",
    "\n",
    "If you run this query, you’ll see that the results will all be around the topic of *“kitten”* or related (cats or pets)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Build your first vector search application"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1 Create a Weaviate cluster and connect to it\n",
    "\n",
    "In order to build your own vector search application with Weaviate, you need a place to start and run the Weaviate instance. You can run Weaviate on you local machine or your cloud provider, or you can run a Weaviate instance using the Weaviate Cluster Service (WCS). If you want to run Weaviate yourself using Docker Compose or Kubernetes, follow the [installation steps in Weaviate's documentation](https://weaviate.io/developers/weaviate/current/getting-started/installation.html) to get a `docker-compose.yml` file and how to start it up. In this tutorial, we are using the WCS to start a Weaviate instance and interact with it using the console.\n",
    "\n",
    "**Exercise 1: Create a WCS account on https://console.semi.technology/.**\n",
    "\n",
    "There are two options of how you can create a cluster once you have an account. You can either create one directly in the Console, or you can use the Weaviate Python Client to create a cluster from your code. We are going for the latter option here. We need to install the Weaviate Python Client in the environment first, and then we can use the WCS method to log in. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "!{sys.executable} -m pip install weaviate-client==3.4.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's import the package and log in with your WCS account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from getpass import getpass # hide password\n",
    "import weaviate # to communicate to the Weaviate instance\n",
    "from weaviate.wcs import WCS"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to authenticate to WCS or Weaviate instance (if Weaviate instance has Authentication enable) we need to create an Authentication object. At the moment it supports two types of authentication credentials: \n",
    "* Password credentials: `weaviate.auth.AuthClientPassword(username='WCS_ACCOUNT_EMAIL', password='WCS_ACCOUNT_PASSWORD')`\n",
    "* Token credentials `weaviate.auth.AuthClientCredentials(client_secret=YOUR_SECRET_TOKEN)`\n",
    "\n",
    "For WCS we will use the Password credentials.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_credentials = weaviate.auth.AuthClientPassword(username=input(\"User name: \"), password=getpass('Password: '))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The my_credentials object contains your credentials so be careful not make it public."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_wcs = WCS(my_credentials)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we connected to WCS, we can `create`, `delete_cluster`, `get_clusters`, `get_cluster_config` and check the status of a cluster with `is_ready` method. You can check additional documentation [here](https://weaviate-python-client.readthedocs.io/en/latest/weaviate.wcs.html).\n",
    "\n",
    "*If you want to check the prototype and docstring of any methods in a notebook, run this command: `object.method?`. You can also use the `help()` function.*<br>\n",
    "Ex: `WCS.is_ready?` or `my_wcs.is_ready?` or `help(WCS.is_ready)`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When creating a Weaviate cluster, we also need to define which *vectorizer* (retriever) module we will use. A list of available modules can be found [here](https://weaviate.io/developers/weaviate/current/modules/index.html). For this tutorial we use a [`text2vec-transformers` module](https://weaviate.io/developers/weaviate/current/retriever-vectorizer-modules/text2vec-transformers.html) to vectorize data and queries. The specific Transformers models is [sentence-transformers-paraphrase-MiniLM-L6-v2](https://huggingface.co/sentence-transformers/paraphrase-MiniLM-L6-v2) and is available on HuggingFace. \n",
    "\n",
    "While this model is a vectorizer (or retriever or encoder) model, which calculates vectors from data, we can extend the search pipeline with a *reader* model. Reader models will be called after a *retriever* model retrieved a list of candidate search results from the query. An example of a reader model is a Question Answering model. This model will try to find an answer to a question posed in the query in the list of candidate data points that were returned by the retriever model. In this tutorial, we will add such a Question Answering model to the pipeline, in this case the pretrained [bert-large-uncased-whole-word-masking-finetuned-squad](https://huggingface.co/bert-large-uncased-whole-word-masking-finetuned-squad) which is also available on HuggingFace. \n",
    "\n",
    "Alternatively, you can choose to use the `text2vec-contextionary` module or one of the other `text2vec-transformers` modules to vectorize text. Keep in mind that the free 'sandbox' Weaviate cluster that you get from WCS runs on CPU only, so Transformer models can be slower to import/vectorize data. `text2vec-contextionary` module vectorize text faster. If you want to experiment with a `text2vec-transformers` module, you are advised to run a Weaviate locally or on your cloud and enable GPU. You can change the `module` config parameter to for example (available models can be found [here](https://weaviate.io/developers/weaviate/current/retriever-vectorizer-modules/text2vec-transformers.html)):\n",
    "\n",
    "If you want to add an additional spellcheck module to the Question Answering module and Transformer Retriever module:\n",
    "\n",
    "```python\n",
    "modules = [{\n",
    "    \"name\": \"text2vec-transformers\", \n",
    "    \"tag\": \"sentence-transformers-paraphrase-MiniLM-L6-v2\"\n",
    "    }, { \n",
    "    \"name\": \"qna-transformers\",\n",
    "    \"tag\": \"bert-large-uncased-whole-word-masking-finetuned-squad\"\n",
    "    }, { \n",
    "    \"name\": \"text-spellcheck\",\n",
    "    \"tag\": \"pyspellchecker-en\"\n",
    "    }]\n",
    "```\n",
    "\n",
    "or use the light-weight contextionary:\n",
    "\n",
    "```python\n",
    "modules = [{\n",
    "    \"name\": \"text2vec-contextionary\",\n",
    "    \"tag\": \"en0.16.0-v1.0.0\"\n",
    "    }]\n",
    "```\n",
    "\n",
    "or use the contextionary in another language than English, for example Dutch:\n",
    "\n",
    "```python\n",
    "modules = [{\n",
    "    \"name\": \"text2vec-contextionary\",\n",
    "    \"tag\": \"nl0.16.0-v1.0.2\"\n",
    "    }]\n",
    "```   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with_auth = False # set to true if you don't want your instance to be publically available\n",
    "cluster_name = None # you can change this to some name you like, if kept to None a random name will be generated\n",
    "modules = [{\n",
    "    \"name\": \"text2vec-transformers\", \n",
    "    \"tag\": \"sentence-transformers-paraphrase-MiniLM-L6-v2\"\n",
    "    }, { \n",
    "    \"name\": \"qna-transformers\",\n",
    "    \"tag\": \"bert-large-uncased-whole-word-masking-finetuned-squad\"\n",
    "    }] \n",
    "\n",
    "weaviate_url = my_wcs.create(cluster_name, with_auth=with_auth, modules=modules, wait_for_completion=True) \n",
    "\n",
    "weaviate_url"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Congratulations! You have your first Weaviate cluster running!**\n",
    "\n",
    "You can check the cluster at the given URL. You can check the meta information at `<weaviate_url>/v1/meta`, it's data schema at `<weaviate_url>/v1/schema` and the data objects at `<weaviate_url>/v1/objects`. You can also see this cluster on your homepage on [console.semi.technology](https://console.semi.technology) (when you are logged in). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We need to connect to the cluster with the Python client in order to create a data schema, import data and query data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client = weaviate.Client(weaviate_url)\n",
    "client.is_ready()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2 Get data and analyze it\n",
    "\n",
    "We set up a Weaviate instance, connected to it and have it ready for requests. Now it is time to take a step back and get some data and analyze it. \n",
    "\n",
    "This step, as for all the machine learning models, is the most important one. Here we have to decide what is relevant, what is important and what data structures/types to use.\n",
    "\n",
    "In this example we are going to use news articles, which are pieces of unstructured textual data. For this we are going to need the `newspaper3k` package:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!{sys.executable} -m pip install newspaper3k"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the next code block we defined a function that retrieves articles from a newspaper. It saves the articles in `objects`, which consists of the article's `title`, `authors`, `title`, `word_count` and a UUID3 `id` generated from the article's URL."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk # it is a dependency of newspaper3k\n",
    "nltk.download('punkt')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import newspaper\n",
    "import uuid\n",
    "import json\n",
    "from tqdm import tqdm\n",
    "\n",
    "def get_articles_from_newspaper(\n",
    "        news_url: str, \n",
    "        max_articles: int=100\n",
    "    ) -> None:\n",
    "    \"\"\"\n",
    "    Download and save newspaper articles as weaviate schemas.\n",
    "    Parameters\n",
    "    ----------\n",
    "    newspaper_url : str\n",
    "        Newspaper title.\n",
    "    \"\"\"\n",
    "    \n",
    "    objects = []\n",
    "    \n",
    "    # Build the actual newspaper    \n",
    "    news_builder = newspaper.build(news_url, memoize_articles=False)\n",
    "    \n",
    "    if max_articles > news_builder.size():\n",
    "        max_articles = news_builder.size()\n",
    "    pbar = tqdm(total=max_articles)\n",
    "    pbar.set_description(f\"{news_url}\")\n",
    "    i = 0\n",
    "    while len(objects) < max_articles and i < news_builder.size():\n",
    "        article = news_builder.articles[i]\n",
    "        try:\n",
    "            article.download()\n",
    "            article.parse()\n",
    "            article.nlp()\n",
    "\n",
    "            if (article.title != '' and \\\n",
    "                article.title is not None and \\\n",
    "                article.summary != '' and \\\n",
    "                article.summary is not None and\\\n",
    "                article.authors):\n",
    "\n",
    "                # create an UUID for the article using its URL\n",
    "                article_id = uuid.uuid3(uuid.NAMESPACE_DNS, article.url)\n",
    "\n",
    "                # create the object\n",
    "                objects.append({\n",
    "                    'id': str(article_id),\n",
    "                    'title': article.title,\n",
    "                    'summary': article.summary,\n",
    "                    'authors': article.authors,\n",
    "                    'word_count': len(article.summary.split())\n",
    "                })\n",
    "                \n",
    "                pbar.update(1)\n",
    "\n",
    "        except:\n",
    "            # something went wrong with getting the article, ignore it\n",
    "            pass\n",
    "        i += 1\n",
    "    pbar.close()\n",
    "    return objects"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 2: Let's download articles from some newspapers.** First, we initialize a data list, then you can add newspapers to it, using the function we defined above, like:\n",
    "`data += get_articles_from_newspaper('http://cnn.com')`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = []\n",
    "\n",
    "# for example: \n",
    "# data += get_articles_from_newspaper('https://www.wired.com/')\n",
    "# add more newspapers if you like! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.3 Make a Weaviate data schema\n",
    "\n",
    "We have some data downloaded. Before we can upload it to Weaviate, we need to create a [data schema](https://weaviate.io/developers/weaviate/current/data-schema/index.html). Adding a Weaviate data schema is not a requirement; if you don’t add a data schema, an automatic schema will be generated from the data that you import. But [defining a data schema yourself](https://weaviate.io/developers/weaviate/current/data-schema/schema-configuration.html) allows you to name classes, properties and relations between data entities yourself, so you have full control over the naming and structure, and also about configuration of modules and data vectorizers. \n",
    "\n",
    "A data schema consist of `classes`. Each `class` has a name and has `properties`. A `property` defines a data value that you can add to a `class`. \n",
    "\n",
    "Let's define a minimal schema for our Newspapers example. There will be three classes: `\"Article\"`, `\"Author\"` and `\"Category\"`. `\"Article\"` is already defined, it is now up to you to define the other two classes with the properties: `\"Author\"` has properties `\"name\"` and `\"wroteArticles\"`, `\"Category\"` has one property `\"name\"`. The schema can be visualized as follows:\n",
    "\n",
    "![Shows a visualization of the data schema with object classes and relations](news-schema.png \"Data schema visualization\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the class schema below we create a class named `Article` with the description `An Article class to store the article summary and its authors`. The description is there to explain the user what this class is about.\n",
    "\n",
    "Also we define 4 properties: \n",
    "* `title` - The title of the article, of type `string` (case sensitive),\n",
    "* `summary` - The summary of the article, of data type `text` (case insensitive), \n",
    "* `wordCount` - The amount of words in the summary of the article, of data type `int`, \n",
    "* `hasAuthor` - The authors of the article, of data type `Author`. The `Author` is NOT a primitive data type; this is how we make cross-references between data items. This way you can link your data objects in-between them and create a relation graph. In this case, we need to define the class `Author` as well. The list of primitive data types can be found [here](https://www.semi.technology/developers/weaviate/current/data-schema/datatypes.html).\n",
    "* `hasCategory` - The category of the article, of data type `Category`. The `Category` is NOT a primitive data type; this is how we make cross-references between data items. This way you can link your data objects in-between them and create a relation graph. In this case, we need to define the class `Category` as well. The list of primitive data types can be found [here](https://www.semi.technology/developers/weaviate/current/data-schema/datatypes.html).\n",
    "\n",
    "**NOTE 1:** The properties should always be in cameCase format and starts with a lowercased word.<br>\n",
    "**NOTE 2:** The property data type is always a list because it can accept more than one data type. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise 3: `\"Article\"` is already defined, it is now up to you to define the other two classes with the properties: `\"Author\"` has properties `\"name\"` and `\"wroteArticles\"`, `\"Category\"` has one property `\"name\"`.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "schema = {\n",
    "    \"classes\": [\n",
    "        {\n",
    "            \"class\": \"Article\", # name of the class\n",
    "            \"description\": \"An Article class to store the article summary and its authors\", # a description of what this class represents\n",
    "            \"properties\": [ # class properties\n",
    "                {\n",
    "                    \"name\": \"title\",\n",
    "                    \"dataType\": [\"string\"],\n",
    "                    \"description\": \"The title of the article\", \n",
    "                },\n",
    "                {\n",
    "                    \"name\": \"summary\",\n",
    "                    \"dataType\": [\"text\"],\n",
    "                    \"description\": \"The summary of the article\",\n",
    "                },\n",
    "                {\n",
    "                    \"name\": \"wordCount\",\n",
    "                    \"dataType\": [\"int\"],\n",
    "                    \"description\": \"The number of words in the article's summary\",\n",
    "                },\n",
    "                {\n",
    "                    \"name\": \"hasAuthors\",\n",
    "                    \"dataType\": [\"Author\"],\n",
    "                    \"description\": \"The authors this article has\",\n",
    "                },\n",
    "                {\n",
    "                    \"name\": \"hasCategory\",\n",
    "                    \"dataType\": [\"Category\"],\n",
    "                    \"description\": \"The category of this article\",\n",
    "                }\n",
    "            ]\n",
    "        }, {\n",
    "            # Write the Author class here\n",
    "        }, {\n",
    "            # Write the Category class here\n",
    "        }\n",
    "\n",
    " ]\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we decided on the data structure, we can tell Weaviate this schema. This can be done by accessing the `schema` attribute of the client. \n",
    "\n",
    "The schema can be created by using the [`.create()` method](https://weaviate-python-client.readthedocs.io/en/latest/weaviate.schema.html#weaviate.schema.Schema.create), this option creates multiple classes at once (useful if you have the whole schema). Alternatively you can use the [`.create_class()` method](https://weaviate-python-client.readthedocs.io/en/latest/weaviate.schema.html#weaviate.schema.Schema.create_class), this option creates only one class per call.\n",
    "\n",
    "Also we can check if a schema is present or if a particular class schema is present with the `.contains()` method.\n",
    "\n",
    "More about schema methods, click [here](https://weaviate-python-client.readthedocs.io/en/latest/weaviate.schema.html#module-weaviate.schema) for the Python docs or [here](https://weaviate.io/developers/weaviate/current/restful-api-references/schema.html) for the general schema docs of Weaviate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.schema.create(schema)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets get the schema from weaviate and look what was created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# helper function\n",
    "def prettify(json_dict): \n",
    "    print(json.dumps(json_dict, indent=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prettify(client.schema.get())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can always delete your entire schema with `client.schema.delete_all()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.4 Vectorize & upload data\n",
    "\n",
    "In the previous steps we have preprocessed our data and have created a Weaviate data schema. Now we're ready to add `Articles` and `Authors`.\n",
    "\n",
    "Importing data to weaviate can be done in 2 different ways.\n",
    "\n",
    "1. Adding object by object iteratively. This ca be done using the `data_object` object attribute of the client. This option requires one REST request per object, thus is slower than importing data in batches.\n",
    "2. Adding objects in batches, with the  `Batch` class. This is efficient if you have large amounts of data to upload. The new class also supports 3 different cases of loading data in batches: a) Manually - the user has the absolute control when and how to add and create batches; b) Auto-create batches when full; c) Auto-create batches using dynamic batching, i.e. the batch size is adjusted every time it is created to avoid any Timeout errors.\n",
    "\n",
    "We will use the second method to upload our data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we define functions `add_article()`, `add_author()` and `add_references()`. In this function we make clear how the data that we have saved in our `data` list will be stored in Weaviate, using the schema we defined in the previous step. \n",
    "\n",
    "**EXERCISE4 : Finish the add_author() and add_references() function with the examples in the same code block.** "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from weaviate.batch import Batch # for the typing purposes\n",
    "from weaviate.util import generate_uuid5\n",
    "\n",
    "\n",
    "def add_article(batch: Batch, article_data: dict) -> str:\n",
    "    \n",
    "    article_object = {\n",
    "        'title': article_data['title'],\n",
    "        'wordCount': article_data['word_count'],\n",
    "        'summary': article_data['summary'].replace('\\n', '') # remove newline character\n",
    "    }\n",
    "    article_id = article_data['id']\n",
    "    \n",
    "    # add article to the batch\n",
    "    batch.add_data_object( \n",
    "        data_object=article_object,\n",
    "        class_name='Article',\n",
    "        uuid=article_id\n",
    "    )\n",
    "    \n",
    "    return article_id\n",
    "\n",
    "def add_author(batch: Batch, author_name: str) -> str:\n",
    "    \n",
    "    author_object = {'name': author_name}\n",
    "\n",
    "    # generate an UUID for the Author\n",
    "    author_id = generate_uuid5(author_name)\n",
    "    \n",
    "    # add author to the batch\n",
    "    # EXERCISE: call here the batch.add_data_object function to add the author to the batch\n",
    "    \n",
    "    return author_id\n",
    "\n",
    "def add_references(batch: Batch, article_id: str, author_id: str)-> None:\n",
    "    # add references to the batch\n",
    "    ## Author -> Article\n",
    "    batch.add_reference(\n",
    "        from_object_uuid=author_id,\n",
    "        from_object_class_name='Author',\n",
    "        from_property_name='wroteArticles',\n",
    "        to_object_uuid=article_id\n",
    "    )\n",
    "    \n",
    "    ## Article -> Author \n",
    "    # EXERCISE: call here the batch.add_reference function to add the article->author reference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we use the `batch()` method to upload the data to Weaviate. We are using a dynamic auto-create batch method. The batch method automatically creates objects and references when a batch is full, which is determined dynamically (depending on the object's size). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "client.batch.configure(batch_size=50, dynamic=True, callback=None)\n",
    "\n",
    "with client.batch as batch:\n",
    "\n",
    "    for i in data:\n",
    "\n",
    "        # add article to the batch\n",
    "        article_id = add_article(batch, i)\n",
    "\n",
    "        for author in i['authors']:\n",
    "\n",
    "            # add author to the batch\n",
    "            author_id = add_author(batch, author)\n",
    "\n",
    "            # add cross references to the batch\n",
    "            add_references(batch, article_id=article_id, author_id=author_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If the data upload is successful, you are now ready to query the data!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.5 Query data\n",
    "\n",
    "If you've successfully ran the previous cells, you now have the newspaper data present in your Weaviate cluster! You are ready to query the data.\n",
    "\n",
    "**Exercise 5: Query Weaviate**. You can do this via 1) the GUI on console.semi.technology or 2) via the python client"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.5.1 Query data using the GUI\n",
    "\n",
    "You can fire GraphQL queries to the dataset via [console.semi.technology](https://console.semi.technology). Log in and connect to the correct cluster on the dashboard via the 'connect' button. Then, go to 'Query' in the left-side menu, which leads to a GraphiQL interface. In the left window you can write queries, and the result will be displayed on the right. You can open the query docs by clicking the button on the top right corner. You can also visit the docs on the website for more information about the queries: https://weaviate.io/developers/weaviate/current/graphql-references/index.html. \n",
    "\n",
    "Now, let's try the following queries. Feel free to adjust them and try out more, these are just some examples!\n",
    "\n",
    "**1. Get a list of article names**\n",
    "```graphql\n",
    "{\n",
    "  Get {\n",
    "    Article {\n",
    "      title\n",
    "    }\n",
    "  }\n",
    "}\n",
    "```\n",
    "\n",
    "**2. Get the number of articles present**\n",
    "```graphql\n",
    "{\n",
    "  Aggregate {\n",
    "    Article {\n",
    "      meta {\n",
    "        count\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "```\n",
    "\n",
    "**3. Get the average, minimum and maximum word count of all articles**\n",
    "```graphql\n",
    "{\n",
    "  Aggregate {\n",
    "    Article {\n",
    "      wordCount {\n",
    "        minimum\n",
    "        mean\n",
    "        maximum\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "```\n",
    "\n",
    "**4. Get article titles and cross-ref authors**\n",
    "```graphql\n",
    "{\n",
    "  Get {\n",
    "    Article {\n",
    "      title\n",
    "      hasAuthors {\n",
    "        ... on Author {\n",
    "          name\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "```\n",
    "\n",
    "**5. Get the title, id and vector of the first article**\n",
    "```graphql\n",
    "{\n",
    "  Get {\n",
    "    Article(limit: 1) {\n",
    "      title\n",
    "      _additional {\n",
    "        id\n",
    "        vector\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "```\n",
    "\n",
    "**6. Semantic search: get articles that are near \"animal\" and return the title, summary and semantic certainty**\n",
    "```graphql\n",
    "{\n",
    "  Get {\n",
    "    Article (\n",
    "      nearText: {\n",
    "        concepts: [\"animal\"]\n",
    "      }\n",
    "    ) {\n",
    "      title\n",
    "      summary\n",
    "      _additional {\n",
    "        certainty\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "```\n",
    "\n",
    "**7. Semantic search: get articles that are near \"music\" and return the title, summary and semantic certainty, if certainty is above 0.8**\n",
    "```graphql\n",
    "{\n",
    "  Get {\n",
    "    Article (\n",
    "      nearText: {\n",
    "        concepts: [\"music\"],\n",
    "        certainty: 0.8\n",
    "      }\n",
    "    ) {\n",
    "      title\n",
    "      summary\n",
    "      _additional {\n",
    "        certainty\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "```\n",
    "\n",
    "**8. You can also combine semantic and scalar filters:**\n",
    "```graphql\n",
    "{\n",
    "  Get {\n",
    "    Article (\n",
    "      where: {\n",
    "        operands: [{\n",
    "          path: [\"hasAuthors\"],\n",
    "          valueInt: 2\n",
    "          operator: GreaterThan\n",
    "        }, {\n",
    "          path: [\"wordCount\"],\n",
    "          valueInt: 100\n",
    "          operator: GreaterThan\n",
    "        }], \n",
    "        operator: And\n",
    "      }\n",
    "      nearText: {\n",
    "        concepts: [\"animals\"]\n",
    "      }\n",
    "    ) {\n",
    "      title\n",
    "      wordCount\n",
    "      hasAuthors {\n",
    "        ... on Author {\n",
    "          name\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "```\n",
    "\n",
    "**9. Use the Question Answering module:**\n",
    "```graphql\n",
    "{\n",
    "  Get {\n",
    "    Article(\n",
    "      ask: {\n",
    "        question: \"Which earplugs should I buy?\",\n",
    "        rerank: true,\n",
    "        properties: [\"summary\"]\n",
    "      }\n",
    "      limit: 1\n",
    "    ) {\n",
    "      title\n",
    "      summary\n",
    "      _additional {\n",
    "        answer {\n",
    "          result\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2.5.2 Query data using python\n",
    "\n",
    "You can use all GraphQL functions also with the Weaviate Python client. For reference, [check the  docs](https://weaviate.io/developers/weaviate/current/graphql-references/index.html). Some examples:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**1. Get a list of article names**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = client.query.get(\"Article\", [\"title\"]).do()\n",
    "prettify(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**2. Get the number of articles present**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = client.query.aggregate(\"Article\") \\\n",
    "    .with_fields('meta { count }') \\\n",
    "    .do()\n",
    "prettify(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**3. Get the average, minimum and maximum word count of all articles**\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = client.query.aggregate(\"Article\") \\\n",
    "    .with_fields(\"wordCount {maximum mean minimum}\") \\\n",
    "    .do()\n",
    "prettify(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**4. Get article titles and cross-ref authors**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"{Get {Article {title hasAuthors {... on Author {name }}}}}\"\n",
    "\n",
    "result = client.query.raw(query)\n",
    "\n",
    "prettify(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**5. Get the title, id and vector of the first article**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = client.query.get(\"Article\", [\"title\", \"_additional {id vector} \"]).with_limit(1).do()\n",
    "prettify(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**6. Semantic search: get articles that are near \"animal\" and return the title, summary and semantic certainty**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nearText = {\n",
    "  \"concepts\": [\"animal\"]\n",
    "}\n",
    "\n",
    "client.query.get(\"Article\", [\"title\", \"summary\", \"_additional {certainty} \"]).with_near_text(nearText).do()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**7. Semantic search: get articles that are near \"music\" and return the title, summary and semantic certainty, if certainty is above 0.8**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nearText = {\n",
    "  \"concepts\": [\"music\"],\n",
    "  \"certainty\": 0.8\n",
    "}\n",
    "\n",
    "client.query.get(\"Article\", [\"title\", \"_additional {certainty} \"]).with_near_text(nearText).do()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**8. You can also combine semantic and scalar filters:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "query = \"\"\"{\n",
    "  Get {\n",
    "    Article (\n",
    "      where: {\n",
    "        operands: [{\n",
    "          path: [\"hasAuthors\"],\n",
    "          valueInt: 2\n",
    "          operator: GreaterThan\n",
    "        }, {\n",
    "          path: [\"wordCount\"],\n",
    "          valueInt: 100\n",
    "          operator: GreaterThan\n",
    "        }], \n",
    "        operator: And\n",
    "      }\n",
    "      nearText: {\n",
    "        concepts: [\"animals\"]\n",
    "      }\n",
    "    ) {\n",
    "      title\n",
    "      wordCount\n",
    "      hasAuthors {\n",
    "        ... on Author {\n",
    "          name\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\"\"\"\n",
    "result = client.query.raw(query)\n",
    "prettify(result)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**9. Use the Question Answering module:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ask = {\n",
    "  \"question\": \"Which earplugs should I buy?\",\n",
    "  \"rerank\": True,\n",
    "  \"properties\": [\"summary\"]\n",
    "}\n",
    "\n",
    "client.query.get(\"Article\", [\"title\", \"_additional {answer {hasAnswer certainty property result startPosition endPosition} }\"]).with_ask(ask).with_limit(1).do()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Congratulations! Now you know how to start a Weaviate cluster and use the Python client to upload and query data! \n",
    "\n",
    "We've set up a very basic vector search application. For more features, check out the documentation. For example, you can add [Question Answering](https://weaviate.io/developers/weaviate/current/reader-generator-modules/qna-transformers.html), do [multi-modal search](https://weaviate.io/developers/weaviate/current/retriever-vectorizer-modules/multi2vec-clip.html), start automatic [classifications](https://weaviate.io/developers/weaviate/current/restful-api-references/classification.html), and much more! \n",
    "\n",
    "You can join the [Weaviate Slack channel](https://join.slack.com/t/weaviate/shared_invite/zt-goaoifjr-o8FuVz9b1HLzhlUfyfddhw) to interact with other Weaviate users and ask questions or give feedback :) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## BONUS: Automatic data classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the schema, we added the class \"Category\", and the property \"hasCategory\" in \"Article\" which points to this class. The articles are not yet assigned to a category, these references are not filled yet. You can check this by doing the following query: \n",
    "```graphql\n",
    "{\n",
    "  Get {\n",
    "    Article {\n",
    "      title \n",
    "      hasCategory {\n",
    "        ... on Category {\n",
    "          name\n",
    "        }\n",
    "      }\n",
    "    }\n",
    "  }\n",
    "}\n",
    "```\n",
    "\n",
    "You will see that the name of the category will be `null` for all articles. \n",
    "\n",
    "We can use zero-shot classification to automatically assign articles to a category. For more information, check [the docs](https://weaviate.io/developers/weaviate/current/restful-api-references/classification.html#zero-shot-classification).\n",
    "\n",
    "**Exercise 6: Zero shot classification to assign categories to articles** \n",
    "\n",
    "Follow the steps below. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, define some categories. You can choose yourself which categories you want to add. Keep in mind that every article will be assigned to one of the categories (no article will be left), with this simple form of classification. Categories can be for example \"Business\", \"Sports\", \"Environment\", etc. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the categories, add as many as you want!\n",
    "categories = [\"Business\", \"Arts\", \"Technology\", \"...\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add the categories as data objects\n",
    "for category in categories:\n",
    "    client.batch.add_data_object({\"name\": category}, \"Category\", generate_uuid5(category))\n",
    "client.batch.create_objects()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# perform the classification\n",
    "client.classification.schedule()\\\n",
    "            .with_type(\"zeroshot\")\\\n",
    "            .with_class_name(\"Article\")\\\n",
    "            .with_classify_properties([\"hasCategory\"])\\\n",
    "            .with_based_on_properties([\"summary\"])\\\n",
    "            .with_wait_for_completion()\\\n",
    "            .do()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once the classification is complete and successful, you can check if your articles now have a category assigned, using a GraphQL query."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "3579617d9edeca879b1c2e9436d575a3656d7c05fdfb8572acef0a8c9e2fafea"
  },
  "kernelspec": {
   "display_name": "Python 3.8.5 64-bit ('3.8.5')",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}