quickstarts/python_quickstart.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "bb2094b8-13a5-4f7c-bd21-d2c709dab914",
   "metadata": {
    "id": "bb2094b8-13a5-4f7c-bd21-d2c709dab914"
   },
   "source": [
    "# Quickstart: Hello, KDB.AI\n",
    "\n",
    "##### Note: This example requires a KDB.AI endpoint and API key. Sign up for a free [KDB.AI account](https://kdb.ai/get-started).\n",
    "\n",
    "How to get started with the KDB.AI vector database. Here, you'll get a quick taste of KDB.AI in  ~10 minutes.\n",
    "\n",
    "You will learn how to:\n",
    "\n",
    "1. Connect to KDB.AI\n",
    "1. Create a KDB.AI Table\n",
    "1. Add Data to the KDB.AI Table\n",
    "1. Query the Table\n",
    "1. Perform Similarity Search\n",
    "1. Delete the KDB.AI Table"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "260d0f4b-ef09-4bd2-a197-a9351be24684",
   "metadata": {
    "id": "260d0f4b-ef09-4bd2-a197-a9351be24684"
   },
   "source": [
    "## 0. Setup"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1468bd3",
   "metadata": {
    "id": "d1468bd3"
   },
   "source": [
    "### Install dependencies\n",
    "\n",
    "In order to successfully run this sample, note the following steps depending on where you are running this notebook:\n",
    "\n",
    "-***Run Locally / Private Environment:*** The [Setup](https://github.com/KxSystems/kdbai-samples/blob/main/README.md#setup) steps in the repository's `README.md` will guide you on prerequisites and how to run this with Jupyter.\n",
    "\n",
    "\n",
    "-***Colab / Hosted Environment:*** Open this notebook in Colab and run through the cells.\n",
    "\n",
    "#### Embedding Library\n",
    "\n",
    "To generate embeddings, we will be using FastEmbed, a fast, lightweight alternative to Sentence Transformers.\n",
    "\n",
    "It supports a variety of popular text models and is built for efficiency and accuracy. In this notebook, we will use FastEmbed to generate embeddings for company descriptions, which we will then store in a KDB.AI table and use for similarity search."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "491cd6d6",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 1000
    },
    "collapsed": true,
    "id": "491cd6d6",
    "outputId": "c126432d-bdcb-4b9e-e964-6f6d8bfd803a"
   },
   "outputs": [],
   "source": [
    "!pip install kdbai_client fastembed onnxruntime==1.19.2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc6d17b7",
   "metadata": {
    "id": "cc6d17b7"
   },
   "source": [
    "### Import Packages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "805d97da",
   "metadata": {
    "id": "805d97da"
   },
   "outputs": [],
   "source": [
    "# vector DB\n",
    "import os\n",
    "from getpass import getpass\n",
    "import kdbai_client as kdbai\n",
    "from fastembed import TextEmbedding\n",
    "import time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "a55ae34e-472b-4aa7-9add-1fcb2ee24a41",
   "metadata": {
    "id": "a55ae34e-472b-4aa7-9add-1fcb2ee24a41"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c660c7d",
   "metadata": {
    "id": "8c660c7d"
   },
   "source": [
    "## 1. Connect to KDB.AI"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3a3aa22",
   "metadata": {
    "id": "d3a3aa22"
   },
   "source": [
    "### Define KDB.AI Session\n",
    "\n",
    "KDB.AI comes in two offerings:\n",
    "\n",
    "1. [KDB.AI Cloud](https://trykdb.kx.com/kdbai/signup/) - For experimenting with smaller generative AI projects with a vector database in our cloud.\n",
    "2. [KDB.AI Server](https://trykdb.kx.com/kdbaiserver/signup/) - For evaluating large scale generative AI applications on-premises or on your own cloud provider.\n",
    "\n",
    "Depending on which you use there will be different setup steps and connection details required.\n",
    "\n",
    "##### Option 1. KDB.AI Cloud\n",
    "\n",
    "To use KDB.AI Cloud, you will need two session details - a URL endpoint and an API key.\n",
    "To get these you can sign up for free [here](https://trykdb.kx.com/kdbai/signup).\n",
    "\n",
    "You can connect to a KDB.AI Cloud session using `kdbai.Session` and passing the session URL endpoint and API key details from your KDB.AI Cloud portal.\n",
    "\n",
    "If the environment variables `KDBAI_ENDPOINTS` and `KDBAI_API_KEY` exist on your system containing your KDB.AI Cloud portal details, these variables will automatically be used to connect.\n",
    "If these do not exist, it will prompt you to enter your KDB.AI Cloud portal session URL endpoint and API key details."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "2e85c1ff",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "collapsed": true,
    "id": "2e85c1ff",
    "outputId": "67920c0b-0832-439f-f887-793b0ad00617"
   },
   "outputs": [],
   "source": [
    "KDBAI_ENDPOINT = (\n",
    "    os.environ[\"KDBAI_ENDPOINT\"]\n",
    "    if \"KDBAI_ENDPOINT\" in os.environ\n",
    "    else input(\"KDB.AI endpoint: \")\n",
    ")\n",
    "KDBAI_API_KEY = (\n",
    "    os.environ[\"KDBAI_API_KEY\"]\n",
    "    if \"KDBAI_API_KEY\" in os.environ\n",
    "    else getpass(\"KDB.AI API key: \")\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6f330098",
   "metadata": {
    "id": "6f330098"
   },
   "outputs": [],
   "source": [
    "session = kdbai.Session(api_key=KDBAI_API_KEY, endpoint=KDBAI_ENDPOINT)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86f16226",
   "metadata": {
    "id": "86f16226"
   },
   "source": [
    "##### Option 2. KDB.AI Server\n",
    "\n",
    "To use KDB.AI Server, you will need download and run your own container.\n",
    "To do this, you will first need to sign up for free [here](https://trykdb.kx.com/kdbaiserver/signup/).\n",
    "\n",
    "You will receive an email with the required license file and bearer token needed to download your instance.\n",
    "Follow instructions in the signup email to get your session up and running.\n",
    "\n",
    "Once the [setup steps](https://code.kx.com/kdbai/gettingStarted/kdb-ai-server-setup.html) are complete you can then connect to your KDB.AI Server session using `kdbai.Session` and passing your local endpoint."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "ff249f70",
   "metadata": {
    "id": "ff249f70"
   },
   "outputs": [],
   "source": [
    "# session = kdbai.Session(endpoint=\"http://localhost:8082\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ec2c77b",
   "metadata": {
    "id": "1ec2c77b"
   },
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "<b>Need help understanding a function?</b><br/>\n",
    "Add ? before or after any function name in KDB.AI to bring up the documentation for that function along with sample code and arguments.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "6e54917b",
   "metadata": {
    "id": "6e54917b"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31mInit signature:\u001b[0m\n",
      "\u001b[0mkdbai\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSession\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mapi_key\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Optional[str]'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;34m*\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mendpoint\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Optional[str]'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'http://localhost:8082'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mhost\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Optional[str]'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mport\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Optional[int]'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mmode\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'str'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'qipc'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0moptions\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Optional[dict]'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;34m'None'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mDocstring:\u001b[0m      Session class to maintain connection to KDB.AI server\n",
      "\u001b[0;31mInit docstring:\u001b[0m\n",
      "Create a connection to a KDB.AI server.\n",
      "\n",
      "Args:\n",
      "    api_key: API Key to be used for authentication.\n",
      "    endpoint: REST server endpoint to connect to.\n",
      "    host: QIPC host to connect to (defaults to endpoint's hostname).\n",
      "    port: QIPC port to connect to (defaults to endpoint's port).\n",
      "    mode: Communication mode betwwen client and server. [qipc|rest]\n",
      "    options: Extra parameters to create connection.\n",
      "\n",
      "Example:\n",
      "    Open a session on KDB.AI Cloud with an api key:\n",
      "\n",
      "    ```python\n",
      "    session = Session(endpoint='YOUR_INSTANCE_ENDPOINT', api_key='YOUR_API_KEY')\n",
      "    ```\n",
      "\n",
      "    Open a session on a custom KDB.AI instance on http://localhost:8082:\n",
      "\n",
      "    ```python\n",
      "    session = Session(endpoint='http://localhost:8082')\n",
      "    ```\n",
      "\n",
      "    Alternatively you can create a session with `host` and `port` parameters:\n",
      "    ```python\n",
      "    session = Session(host='localhost', port=8082, endpoint=None)\n",
      "    ```\n",
      "\n",
      "    By default the session will use QIPC connection except for cloud instances where api key is necessary.\n",
      "    You can enforce using REST requeset by providing `mode='rest'`\n",
      "    ```\n",
      "    session = Session(endpoint='http://localhost:8082', mode='rest')\n",
      "    ```\n",
      "\u001b[0;31mFile:\u001b[0m           ~/kx/similarity-search/kdbai-python-client/src/kdbai_client/api.py\n",
      "\u001b[0;31mType:\u001b[0m           type\n",
      "\u001b[0;31mSubclasses:\u001b[0m     "
     ]
    }
   ],
   "source": [
    "?kdbai.Session"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2b41f7c",
   "metadata": {},
   "source": [
    "### Verify Defined Databases\n",
    "\n",
    "We can check our connection using the `session.databases()` function.\n",
    "This will return a list of all the databases we have defined in our vector database thus far.\n",
    "This should return a \"default\" database along with any other databases you have already created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "dee8fd32",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[KDBAI database \"default\"]"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session.databases()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8788a6b1",
   "metadata": {
    "id": "8788a6b1"
   },
   "source": [
    "### Verify Defined Tables\n",
    "\n",
    "We can check our connection using the `session.list()` function.\n",
    "This will return a list of all the tables we have defined in our vector database thus far.\n",
    "This should return an empty list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "97e5f4a9",
   "metadata": {
    "id": "97e5f4a9"
   },
   "outputs": [],
   "source": [
    "# ensure no table called \"company_data\" exists\n",
    "try:\n",
    "    for t in session.database('default').tables:\n",
    "            if t.name == 'company_data':\n",
    "                t.drop() \n",
    "    time.sleep(5)\n",
    "except kdbai.KDBAIException:\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "884f1b49-7f03-419b-8f10-2086fcc4d0fb",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "884f1b49-7f03-419b-8f10-2086fcc4d0fb",
    "outputId": "3ad10654-45bb-4f50-e397-6e3a2f81f442"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "session.database('default').tables"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e33f03c3",
   "metadata": {
    "id": "e33f03c3"
   },
   "source": [
    "## 2. Create a KDB.AI Table\n",
    "\n",
    "To create a table we can use `create_table`, this function takes two arguments - the name and schema of the table.\n",
    "\n",
    "This schema must meet the following criteria:\n",
    "- It must contain a list of columns.\n",
    "- All columns must have either a `type` or a `qtype` specified, except the column of vectors.\n",
    "- One column of vector embeddings may also have a `vectorIndex` attribute with the configuration of the index for similarity search - this column is implicitly an array of `float64`.\n",
    "\n",
    "Run `?session.database('default').create_table` for more details and sample code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "fnOlGAnN6uqa",
   "metadata": {
    "id": "fnOlGAnN6uqa"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31mSignature:\u001b[0m\n",
      "\u001b[0mdatabase\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcreate_table\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mtable\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'str'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mschema\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Optional[List[Dict[str, Any]]]'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mindexes\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'List[Dict[str, Any]]'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m[\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mpartition_column\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Optional[str]'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0membedding_configurations\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Optional[Dict[str, Any]]'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mexternal_data_references\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Optional[List[Dict[str, Any]]]'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mdefault_result_type\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'str'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'pd'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m->\u001b[0m \u001b[0;34m'Table'\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mDocstring:\u001b[0m\n",
      "Create a table with a schema\n",
      "\n",
      "Args:\n",
      "    table: Name of the table to create.\n",
      "    schema: list of dictionaries containing column name and type\n",
      "    indexes: list of dictionaries containing index definitions (multiple indexes can be created for each column)\n",
      "    partition_column: column name if table is partitioned\n",
      "    embedding_configurations: dictionary containing TSC configurations\n",
      "    external_data_references: reference info of existing table\n",
      "    default_result_type: default result type for search and query methods (pd|py|q)\n",
      "\n",
      "Returns:\n",
      "    A newly created `Table` object based on parameters.\n",
      "\n",
      "Raises:\n",
      "    KDBAIException: Raised when a error happens during the creation of the table.\n",
      "\n",
      "Example Flat/qFlat Index:\n",
      "    ```python\n",
      "    schema = [\n",
      "        {'name': 'id', 'type': 'int32'},\n",
      "        {'name': 'embeddings', 'type': 'float32s'}\n",
      "    ]\n",
      "    index = {'name': 'flat_index', 'column': 'embeddings', 'type': 'flat',\n",
      "             'params': {'dims': 25, 'metric': 'L2'}}\n",
      "    table = session.create_table('documents', schema=schema, indexes=[index])\n",
      "    ```\n",
      "\n",
      "Example IVF Index:\n",
      "    ```python\n",
      "    schema = [\n",
      "        {'name': 'id', 'type': 'int32'},\n",
      "        {'name': 'embeddings', 'type': 'float32s'}\n",
      "    ]\n",
      "    index = {'name': 'ivf_index', 'column': 'embeddings', 'type': 'ivf',\n",
      "             'params': {'metric': 'L2', 'nclusters': 8}}\n",
      "    table = session.create_table('documents', schema=schema, indexes=[index])\n",
      "    ```\n",
      "\n",
      "Example IVFPQ Index:\n",
      "    ```python\n",
      "    schema = [\n",
      "        {'name': 'id', 'type': 'int32'},\n",
      "        {'name': 'embeddings', 'type': 'float32s'}\n",
      "    ]\n",
      "    index = {{'name': 'ivf_index', 'column': 'embeddings', 'type': 'ivfpq',\n",
      "              'params': {'metric': 'L2', 'nclusters': 8, 'nbits': 8, 'nsplits': 5}}\n",
      "    table = session.create_table('documents', schema=schema, indexes=[index])\n",
      "    ```\n",
      "\n",
      "Example HNSW Index:\n",
      "    ```python\n",
      "    schema = [\n",
      "        {'name': 'id', 'type': 'int32'},\n",
      "        {'name': 'embeddings', 'type': 'float32s'}\n",
      "    ]\n",
      "    index = {'name': 'hnsw_index', 'column': 'embeddings', 'type': 'hnsw',\n",
      "             'params': {'dims': 25, 'metric': 'L2', 'M': 8, 'efConstruction': 8}}\n",
      "    table = session.create_table('documents', schema=schema, indexes=[index])\n",
      "    ```\n",
      "\n",
      "Example Sparse Index:\n",
      "    ```python\n",
      "    schema = [\n",
      "        {'name': 'id', 'type': 'int32'},\n",
      "        {'name': 'embeddings', 'type': 'float32s'}\n",
      "    ]\n",
      "    index = {'name': 'sparse_index', 'column': 'sparse_col', 'type': 'bm25',\n",
      "             'params': {'k': 1.25, 'b': 0.75}}\n",
      "    table = session.create_table('documents', schema=schema, indexes=[index])\n",
      "    ```\n",
      "\n",
      "Example Flat with Sparse Index:\n",
      "    ```python\n",
      "    schema = [\n",
      "        {'name': 'id', 'type': 'int32'},\n",
      "        {'name': 'embeddings', 'type': 'float32s'},\n",
      "        {'name': 'sparse_col', 'type': 'general'}\n",
      "    ]\n",
      "    index_flat = {'name': 'flat_index', 'column': 'embeddings', 'type': 'flat',\n",
      "                  'params': {'dims': 25, 'metric': 'L2'}}\n",
      "    index_sparse = {'name': 'sparse_index', 'column': 'sparse_col', 'type': 'bm25',\n",
      "                    'params': {'k': 1.25, 'b': 0.75}}\n",
      "    table = session.create_table('documents', schema=schema, indexes=[index_flat, index_sparse])\n",
      "    ```\n",
      "\n",
      "Examle Flat with TSC:\n",
      "    ```python\n",
      "    schema = [\n",
      "        {'name': 'id', 'type': 'int32'},\n",
      "        {'name': 'embeddings', 'type': 'float32s'}\n",
      "    ]\n",
      "    index_flat = {'name': 'flat_index', 'column': 'embeddings', 'type': 'flat',\n",
      "                  'params': {'dims': 25, 'metric': 'L2'}}\n",
      "    embedding_conf = {'embeddings': {\"dims\": 4, \"type\": \"tsc\", \"on_insert_error\": \"reject_all\" }}\n",
      "    table = session.create_table('documents',\n",
      "                                 schema=schema,\n",
      "                                 indexes=[index_flat],\n",
      "                                 embedding_configurations=embedding_conf)\n",
      "    ```\n",
      "\u001b[0;31mFile:\u001b[0m      ~/kx/similarity-search/kdbai-python-client/src/kdbai_client/api.py\n",
      "\u001b[0;31mType:\u001b[0m      method"
     ]
    }
   ],
   "source": [
    "database = session.database('default')\n",
    "?database.create_table"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9da55253",
   "metadata": {
    "id": "9da55253"
   },
   "source": [
    "### Define Schema\n",
    "\n",
    "Our table will have three columns: the first two, `company_name` `company_description` contain company names and descriptions, the third will be the vector embeddings we will use for similarity search later on in this example.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "e5e8b782",
   "metadata": {
    "id": "e5e8b782"
   },
   "outputs": [],
   "source": [
    "schema = [\n",
    "        {\"name\": \"company_name\", \"type\": \"str\"},\n",
    "        {\"name\": \"company_description\", \"type\": \"str\"},\n",
    "        {\"name\": \"vectors\", \"type\": \"float64s\"}\n",
    "    ]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53b0354d",
   "metadata": {},
   "source": [
    "### Define the indexes\n",
    "We will define our dimensionality, similarity metric and index type with the vectorIndex attribute. For this example we chose:\n",
    "\n",
    "- type = flat : You have the choice of using other indexes like, hnsw, qHNSW, and IVFPQ, or a qFlat index here, as with metrics the one you chose depends your data and your overall performance requirements.\n",
    "- name = vectorIndex : this is a custom name you give your index.\n",
    "- column = vectors : this is the column where the embeddings are stored.\n",
    "#### params:\n",
    "- dims = 384 : In the next section, we generate embeddings that are 384-dimensional to match this. The number of dimensions should mirror the output dimensions of your embedding model.\n",
    "- metric = CS : We chose Cosine Similarity. You have the choice of using other metrics here like IP/Inner Product and CS/Cosine Similarity and the one you chose depends on the specific context and nature of your data.\n",
    "!Note, it is possible to define multiple indexes within a table!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "0a17fc2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define the index\n",
    "indexes = [\n",
    "            {\n",
    "                \"name\": \"vectorIndex\", \"type\": \"flat\", \n",
    "                \"params\": {\"dims\": 384, \"metric\": \"CS\"},\n",
    "                \"column\": \"vectors\"\n",
    "            }\n",
    "        ]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09a5caa0",
   "metadata": {
    "id": "09a5caa0"
   },
   "source": [
    "### Create Table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "dXfzhaTCGoH_",
   "metadata": {
    "id": "dXfzhaTCGoH_"
   },
   "outputs": [],
   "source": [
    "database = session.database('default')\n",
    "table = database.create_table(\"company_data\", schema, indexes=indexes)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20afbea1",
   "metadata": {
    "id": "20afbea1"
   },
   "source": [
    "## 3. Add Data to the KDB.AI Table\n",
    "\n",
    "First, let's define a list of companies and their descriptions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "37581e86",
   "metadata": {
    "id": "37581e86"
   },
   "outputs": [],
   "source": [
    "company_data = [\n",
    "    (\"Apple\", \"A technology company known for its iPhones, MacBooks, and innovative designs\"),\n",
    "    (\"Google\", \"A search engine giant that also specializes in advertising, cloud computing, and AI\"),\n",
    "    (\"Brave\", \"A privacy-focused search engine and browser.\"),\n",
    "    (\"Perplexity\", \"An answer engine that searches the internet and uses a large language model to summarize web data.\"),\n",
    "    (\"Amazon\", \"An e-commerce leader that offers a wide range of products and services, including AWS\"),\n",
    "    (\"Microsoft\", \"A technology company known for its software products like Windows and Office\"),\n",
    "    (\"Facebook\", \"A social media platform that connects people worldwide and owns Instagram and WhatsApp\"),\n",
    "    (\"Tesla\", \"An electric vehicle manufacturer known for its innovative and sustainable energy solutions\"),\n",
    "    (\"Rivian\", \"An electric vehicle company focusing on adventure-oriented trucks and SUVs\"),\n",
    "    (\"Lucid Motors\", \"A company specializing in high-performance electric luxury vehicles\"),\n",
    "    (\"Netflix\", \"A streaming service that offers a wide variety of TV shows, movies, and original content\"),\n",
    "    (\"Hulu\", \"A streaming platform providing a wide range of TV shows, movies, and original content\"),\n",
    "    (\"Disney+\", \"A streaming service offering movies, TV shows, and original content from Disney\"),\n",
    "    (\"Uber\", \"A ride-sharing company that also offers food delivery and freight services\"),\n",
    "    (\"Lyft\", \"A ride-sharing platform connecting passengers with drivers\"),\n",
    "    (\"Didi\", \"A Chinese ride-sharing company offering various transportation services\"),\n",
    "    (\"Airbnb\", \"A platform that allows people to rent out their homes or find lodging worldwide\"),\n",
    "    (\"Vrbo\", \"A vacation rental online marketplace where homeowners list their properties for short-term rentals\"),\n",
    "    (\"Booking.com\", \"An online travel agency offering lodging reservations and other travel products\"),\n",
    "    (\"Spotify\", \"A music streaming service offering a wide range of songs, albums, and podcasts\"),\n",
    "    (\"Apple Music\", \"A music and video streaming service developed by Apple Inc.\"),\n",
    "    (\"YouTube Music\", \"A music streaming service developed by YouTube\"),\n",
    "    (\"Twitter\", \"A social media platform for sharing short messages and real-time updates\"),\n",
    "    (\"Instagram\", \"A photo and video sharing social networking service\"),\n",
    "    (\"Snapchat\", \"A multimedia messaging app known for its disappearing messages\"),\n",
    "    (\"LinkedIn\", \"A professional networking platform for job seekers and employers\"),\n",
    "    (\"Slack\", \"A collaboration platform for team communication and project management\"),\n",
    "    (\"Microsoft Teams\", \"A collaboration platform for team communication and project management\"),\n",
    "    (\"Zoom\", \"A video conferencing platform used for virtual meetings and webinars\")\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e97ace30-d178-4911-a147-0e2a733ac9c3",
   "metadata": {
    "id": "e97ace30-d178-4911-a147-0e2a733ac9c3"
   },
   "source": [
    "Now let's define an embedding model. Here we are using the default embedding model in FastEmbed, BAAI/bge-small-en-v1.5, which has 384 dimensions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c75ed61e-7b6b-4614-90be-20bb8e6355e0",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 49,
     "referenced_widgets": [
      "c873cf1d86a642fbba9c5c3714ab279c",
      "7223e3a082da40c0bfc607a2970d0acc",
      "3dd7e95da10d4c9a987fb90ffccfd7ce",
      "7c590f17f5304259855eef5e727ef280",
      "d3ca822343ad490b9054cf08c913c48b",
      "7e3bed2b82524fd083b76926897b9f18",
      "77fd1cceeb0f4087909e08daa64a0f06",
      "30429cd24ff1429b9c4efafe0a95aff5",
      "235128a18a4c47459047d47a32c61ebe",
      "dd099525ffcb48bdbad60b52b8d1307f",
      "93c2fb60dd1b4222bafa6df44af8264d"
     ]
    },
    "collapsed": true,
    "id": "c75ed61e-7b6b-4614-90be-20bb8e6355e0",
    "outputId": "e0fb5a6f-71de-48b4-af06-b1a1dbab28ce"
   },
   "outputs": [],
   "source": [
    "embedding_model = TextEmbedding()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7831d5a4-99bd-4859-8686-516c8e0a3110",
   "metadata": {
    "id": "7831d5a4-99bd-4859-8686-516c8e0a3110"
   },
   "source": [
    "Let's generate an embedding for each company. Because embedding_model.embed returns a generator, we turn it into a list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "f5dc41e8",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "f5dc41e8",
    "outputId": "1cf0b9b4-e31a-46d3-b8a8-ca458822bd2c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "384"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Example ID values\n",
    "embeddings = list(embedding_model.embed([desc for _, desc in company_data]))\n",
    "len(embeddings[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ba8366f2-172d-48e6-a24e-20a303b1295b",
   "metadata": {
    "id": "ba8366f2-172d-48e6-a24e-20a303b1295b"
   },
   "source": [
    "##### Create a dataframe from our company data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "4a31f878",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "4a31f878",
    "outputId": "6ae1c292-79dd-40ce-bab0-760a6db5858d"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>company_name</th>\n",
       "      <th>company_description</th>\n",
       "      <th>vectors</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Apple</td>\n",
       "      <td>A technology company known for its iPhones, Ma...</td>\n",
       "      <td>[-0.034876905, 0.032589626, -0.002934602, -0.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Google</td>\n",
       "      <td>A search engine giant that also specializes in...</td>\n",
       "      <td>[-0.017866805, -0.057211027, -0.028582964, 0.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Brave</td>\n",
       "      <td>A privacy-focused search engine and browser.</td>\n",
       "      <td>[-0.017717587, -0.020544883, -0.024149919, -0....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Perplexity</td>\n",
       "      <td>An answer engine that searches the internet an...</td>\n",
       "      <td>[-0.043428417, 0.0026718834, -0.014964736, 0.0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Amazon</td>\n",
       "      <td>An e-commerce leader that offers a wide range ...</td>\n",
       "      <td>[-0.04374583, -0.05412757, 0.03689078, -0.0363...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  company_name                                company_description  \\\n",
       "0        Apple  A technology company known for its iPhones, Ma...   \n",
       "1       Google  A search engine giant that also specializes in...   \n",
       "2        Brave       A privacy-focused search engine and browser.   \n",
       "3   Perplexity  An answer engine that searches the internet an...   \n",
       "4       Amazon  An e-commerce leader that offers a wide range ...   \n",
       "\n",
       "                                             vectors  \n",
       "0  [-0.034876905, 0.032589626, -0.002934602, -0.0...  \n",
       "1  [-0.017866805, -0.057211027, -0.028582964, 0.0...  \n",
       "2  [-0.017717587, -0.020544883, -0.024149919, -0....  \n",
       "3  [-0.043428417, 0.0026718834, -0.014964736, 0.0...  \n",
       "4  [-0.04374583, -0.05412757, 0.03689078, -0.0363...  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "names = [company for company, _ in company_data]\n",
    "descriptions = [description for _, description in company_data]\n",
    "\n",
    "# column names/types matching the schema\n",
    "embeddings_df = pd.DataFrame({\"company_name\": names, \"company_description\": descriptions, \"vectors\": list(embeddings)})\n",
    "embeddings_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43cd2ad8",
   "metadata": {
    "id": "43cd2ad8"
   },
   "source": [
    "We can now add data to our KDB.AI table using `insert`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "b7e0f8c5",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 35
    },
    "id": "b7e0f8c5",
    "outputId": "49944585-b4ac-4242-9f29-1332e36ad24e"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'rowsInserted': 29}"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table.insert(embeddings_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09577e8e",
   "metadata": {
    "id": "09577e8e"
   },
   "source": [
    "## 4. Query the Table\n",
    "\n",
    "We can use `query` to query data from the table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "f4b8b8e5",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 959
    },
    "id": "f4b8b8e5",
    "outputId": "5a906745-c9b9-439e-c02b-2f251b07b1e9"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>company_name</th>\n",
       "      <th>company_description</th>\n",
       "      <th>vectors</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Apple</td>\n",
       "      <td>A technology company known for its iPhones, Ma...</td>\n",
       "      <td>[-0.034876905381679535, 0.0325896255671978, -0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Google</td>\n",
       "      <td>A search engine giant that also specializes in...</td>\n",
       "      <td>[-0.01786680519580841, -0.057211026549339294, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Brave</td>\n",
       "      <td>A privacy-focused search engine and browser.</td>\n",
       "      <td>[-0.01771758683025837, -0.020544882863759995, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Perplexity</td>\n",
       "      <td>An answer engine that searches the internet an...</td>\n",
       "      <td>[-0.043428417295217514, 0.0026718834415078163,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Amazon</td>\n",
       "      <td>An e-commerce leader that offers a wide range ...</td>\n",
       "      <td>[-0.04374583065509796, -0.05412757024168968, 0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Microsoft</td>\n",
       "      <td>A technology company known for its software pr...</td>\n",
       "      <td>[-0.03227386251091957, 0.018396837636828423, 0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Facebook</td>\n",
       "      <td>A social media platform that connects people w...</td>\n",
       "      <td>[0.001751603209413588, -0.07014184445142746, 0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Tesla</td>\n",
       "      <td>An electric vehicle manufacturer known for its...</td>\n",
       "      <td>[-0.0014115246012806892, 0.07673310488462448, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Rivian</td>\n",
       "      <td>An electric vehicle company focusing on advent...</td>\n",
       "      <td>[-0.004529878031462431, 0.05161484703421593, 0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Lucid Motors</td>\n",
       "      <td>A company specializing in high-performance ele...</td>\n",
       "      <td>[0.003517858451232314, 0.07283230125904083, 0....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>Netflix</td>\n",
       "      <td>A streaming service that offers a wide variety...</td>\n",
       "      <td>[-0.06678618490695953, -0.04583403840661049, 0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>Hulu</td>\n",
       "      <td>A streaming platform providing a wide range of...</td>\n",
       "      <td>[-0.05717369541525841, -0.06787450611591339, 0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>Disney+</td>\n",
       "      <td>A streaming service offering movies, TV shows,...</td>\n",
       "      <td>[-0.08020051568746567, -0.06656964868307114, 0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>Uber</td>\n",
       "      <td>A ride-sharing company that also offers food d...</td>\n",
       "      <td>[-0.0012249506544321775, -0.06182726100087166,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>Lyft</td>\n",
       "      <td>A ride-sharing platform connecting passengers ...</td>\n",
       "      <td>[-0.01856295019388199, -0.02381841652095318, 0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>Didi</td>\n",
       "      <td>A Chinese ride-sharing company offering variou...</td>\n",
       "      <td>[-0.025807108730077744, -0.03256842494010925, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>Airbnb</td>\n",
       "      <td>A platform that allows people to rent out thei...</td>\n",
       "      <td>[-0.0020053349435329437, -0.0806913897395134, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>Vrbo</td>\n",
       "      <td>A vacation rental online marketplace where hom...</td>\n",
       "      <td>[-0.023459283635020256, -0.017960241064429283,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>Booking.com</td>\n",
       "      <td>An online travel agency offering lodging reser...</td>\n",
       "      <td>[-0.011170933023095131, -0.01784929819405079, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>Spotify</td>\n",
       "      <td>A music streaming service offering a wide rang...</td>\n",
       "      <td>[-0.04392420873045921, -0.08940696716308594, 0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>Apple Music</td>\n",
       "      <td>A music and video streaming service developed ...</td>\n",
       "      <td>[-0.03607852756977081, -0.07780104875564575, -...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21</th>\n",
       "      <td>YouTube Music</td>\n",
       "      <td>A music streaming service developed by YouTube</td>\n",
       "      <td>[-0.044059764593839645, -0.06850195676088333, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22</th>\n",
       "      <td>Twitter</td>\n",
       "      <td>A social media platform for sharing short mess...</td>\n",
       "      <td>[-0.07155980169773102, 0.019955234602093697, -...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23</th>\n",
       "      <td>Instagram</td>\n",
       "      <td>A photo and video sharing social networking se...</td>\n",
       "      <td>[-0.03469916060566902, -0.016102692112326622, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24</th>\n",
       "      <td>Snapchat</td>\n",
       "      <td>A multimedia messaging app known for its disap...</td>\n",
       "      <td>[-0.0604371502995491, 0.03670385479927063, -0....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25</th>\n",
       "      <td>LinkedIn</td>\n",
       "      <td>A professional networking platform for job see...</td>\n",
       "      <td>[-0.08664378523826599, 0.012671931646764278, -...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>26</th>\n",
       "      <td>Slack</td>\n",
       "      <td>A collaboration platform for team communicatio...</td>\n",
       "      <td>[-0.04067962244153023, 0.02870851941406727, -0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>27</th>\n",
       "      <td>Microsoft Teams</td>\n",
       "      <td>A collaboration platform for team communicatio...</td>\n",
       "      <td>[-0.04067962244153023, 0.02870851941406727, -0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>28</th>\n",
       "      <td>Zoom</td>\n",
       "      <td>A video conferencing platform used for virtual...</td>\n",
       "      <td>[-0.047964468598365784, -0.001985185779631138,...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       company_name                                company_description  \\\n",
       "0             Apple  A technology company known for its iPhones, Ma...   \n",
       "1            Google  A search engine giant that also specializes in...   \n",
       "2             Brave       A privacy-focused search engine and browser.   \n",
       "3        Perplexity  An answer engine that searches the internet an...   \n",
       "4            Amazon  An e-commerce leader that offers a wide range ...   \n",
       "5         Microsoft  A technology company known for its software pr...   \n",
       "6          Facebook  A social media platform that connects people w...   \n",
       "7             Tesla  An electric vehicle manufacturer known for its...   \n",
       "8            Rivian  An electric vehicle company focusing on advent...   \n",
       "9      Lucid Motors  A company specializing in high-performance ele...   \n",
       "10          Netflix  A streaming service that offers a wide variety...   \n",
       "11             Hulu  A streaming platform providing a wide range of...   \n",
       "12          Disney+  A streaming service offering movies, TV shows,...   \n",
       "13             Uber  A ride-sharing company that also offers food d...   \n",
       "14             Lyft  A ride-sharing platform connecting passengers ...   \n",
       "15             Didi  A Chinese ride-sharing company offering variou...   \n",
       "16           Airbnb  A platform that allows people to rent out thei...   \n",
       "17             Vrbo  A vacation rental online marketplace where hom...   \n",
       "18      Booking.com  An online travel agency offering lodging reser...   \n",
       "19          Spotify  A music streaming service offering a wide rang...   \n",
       "20      Apple Music  A music and video streaming service developed ...   \n",
       "21    YouTube Music     A music streaming service developed by YouTube   \n",
       "22          Twitter  A social media platform for sharing short mess...   \n",
       "23        Instagram  A photo and video sharing social networking se...   \n",
       "24         Snapchat  A multimedia messaging app known for its disap...   \n",
       "25         LinkedIn  A professional networking platform for job see...   \n",
       "26            Slack  A collaboration platform for team communicatio...   \n",
       "27  Microsoft Teams  A collaboration platform for team communicatio...   \n",
       "28             Zoom  A video conferencing platform used for virtual...   \n",
       "\n",
       "                                              vectors  \n",
       "0   [-0.034876905381679535, 0.0325896255671978, -0...  \n",
       "1   [-0.01786680519580841, -0.057211026549339294, ...  \n",
       "2   [-0.01771758683025837, -0.020544882863759995, ...  \n",
       "3   [-0.043428417295217514, 0.0026718834415078163,...  \n",
       "4   [-0.04374583065509796, -0.05412757024168968, 0...  \n",
       "5   [-0.03227386251091957, 0.018396837636828423, 0...  \n",
       "6   [0.001751603209413588, -0.07014184445142746, 0...  \n",
       "7   [-0.0014115246012806892, 0.07673310488462448, ...  \n",
       "8   [-0.004529878031462431, 0.05161484703421593, 0...  \n",
       "9   [0.003517858451232314, 0.07283230125904083, 0....  \n",
       "10  [-0.06678618490695953, -0.04583403840661049, 0...  \n",
       "11  [-0.05717369541525841, -0.06787450611591339, 0...  \n",
       "12  [-0.08020051568746567, -0.06656964868307114, 0...  \n",
       "13  [-0.0012249506544321775, -0.06182726100087166,...  \n",
       "14  [-0.01856295019388199, -0.02381841652095318, 0...  \n",
       "15  [-0.025807108730077744, -0.03256842494010925, ...  \n",
       "16  [-0.0020053349435329437, -0.0806913897395134, ...  \n",
       "17  [-0.023459283635020256, -0.017960241064429283,...  \n",
       "18  [-0.011170933023095131, -0.01784929819405079, ...  \n",
       "19  [-0.04392420873045921, -0.08940696716308594, 0...  \n",
       "20  [-0.03607852756977081, -0.07780104875564575, -...  \n",
       "21  [-0.044059764593839645, -0.06850195676088333, ...  \n",
       "22  [-0.07155980169773102, 0.019955234602093697, -...  \n",
       "23  [-0.03469916060566902, -0.016102692112326622, ...  \n",
       "24  [-0.0604371502995491, 0.03670385479927063, -0....  \n",
       "25  [-0.08664378523826599, 0.012671931646764278, -...  \n",
       "26  [-0.04067962244153023, 0.02870851941406727, -0...  \n",
       "27  [-0.04067962244153023, 0.02870851941406727, -0...  \n",
       "28  [-0.047964468598365784, -0.001985185779631138,...  "
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table.query()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "55e69a5b",
   "metadata": {
    "id": "55e69a5b"
   },
   "source": [
    "The `query` function accepts a wide range of arguments to make it easy to filter, aggregate, and sort.\n",
    "Run `?table.query` to see them all.\n",
    "\n",
    "Let's filter for companies starting with the letter 'A' using the 'like' operator. Four rows are returned as expected."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "41c6f156",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 175
    },
    "id": "41c6f156",
    "outputId": "7491f14e-5005-4b8c-dc36-97f2a31528d4"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>company_name</th>\n",
       "      <th>company_description</th>\n",
       "      <th>vectors</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Apple</td>\n",
       "      <td>A technology company known for its iPhones, Ma...</td>\n",
       "      <td>[-0.034876905381679535, 0.0325896255671978, -0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Amazon</td>\n",
       "      <td>An e-commerce leader that offers a wide range ...</td>\n",
       "      <td>[-0.04374583065509796, -0.05412757024168968, 0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Airbnb</td>\n",
       "      <td>A platform that allows people to rent out thei...</td>\n",
       "      <td>[-0.0020053349435329437, -0.0806913897395134, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Apple Music</td>\n",
       "      <td>A music and video streaming service developed ...</td>\n",
       "      <td>[-0.03607852756977081, -0.07780104875564575, -...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  company_name                                company_description  \\\n",
       "0        Apple  A technology company known for its iPhones, Ma...   \n",
       "1       Amazon  An e-commerce leader that offers a wide range ...   \n",
       "2       Airbnb  A platform that allows people to rent out thei...   \n",
       "3  Apple Music  A music and video streaming service developed ...   \n",
       "\n",
       "                                             vectors  \n",
       "0  [-0.034876905381679535, 0.0325896255671978, -0...  \n",
       "1  [-0.04374583065509796, -0.05412757024168968, 0...  \n",
       "2  [-0.0020053349435329437, -0.0806913897395134, ...  \n",
       "3  [-0.03607852756977081, -0.07780104875564575, -...  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "table.query(filter=[(\"like\", \"company_name\", \"A*\")])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c267a58",
   "metadata": {
    "id": "9c267a58"
   },
   "source": [
    "## 5. Perform Similarity Search\n",
    "\n",
    "Finally, let's perform similarity search on the table. We do this using the `search` function."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "F35xiMwy69HI",
   "metadata": {
    "id": "F35xiMwy69HI"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\u001b[0;31mSignature:\u001b[0m\n",
      "\u001b[0mtable\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mvectors\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Dict[str, Any]'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mn\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'int'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0;34m*\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mtype\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Optional[str]'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mindex_params\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Optional[Dict[str, Any]]'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0moptions\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Optional[Dict[str, Any]]'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mfilter\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Optional[List[List[Any]]]'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0msort_columns\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Optional[List[str]]'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mgroup_by\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Optional[List[str]]'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0maggs\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Optional[Dict[str, Any]]'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m    \u001b[0mresult_type\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0;34m'Optional[str]'\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\n",
      "\u001b[0;34m\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mDocstring:\u001b[0m\n",
      "Perform similarity search on the table, supports dense or sparse queries.\n",
      "\n",
      "Args:\n",
      "    vectors: Query vectors for the search. Dictionary keys must be index names or column name(tss) to execute\n",
      "        the search on. Values must be list of vectors containing the search vectors\n",
      "    n: Number of neighbours to return.\n",
      "    type: Override basic similarity search type.\n",
      "    index_params: Index specific options for similarity search.\n",
      "    options: Additional options for search, e.g: renaming distance column.\n",
      "    filter: A list of filter conditions as triplets in the following format:\n",
      "        `[['function', 'column name', 'parameter'], ... ]`\n",
      "        See all filter operators [here](https://code.kx.com/kdbai/use/filter.html#supported-filter-functions)\n",
      "    sort_columns: List of column names to sort on.\n",
      "    group_by: A list of column names to use for group by.\n",
      "    aggs: Dictionary specifying aggregation/projection on table.\n",
      "        If we want to return subset of columns, specify column name as key and values.\n",
      "        If we want to aggregate a column, key must be target column name and value mmust be agg function\n",
      "        and source column\n",
      "        See all aggregation functions [here](https://code.kx.com/kdbai/use/query.html#supported-aggregations)\n",
      "    result_type: data type to convert result into (pd|py|q)\n",
      "\n",
      "Returns:\n",
      "    List tables with one table of matching neighbors for each query vector.\n",
      "\n",
      "Examples:\n",
      "    ```python\n",
      "    #Find the closest neighbour of a single (dense) query vector\n",
      "    table.search(vectors={'my_index': [[0,0,0,0,0,0,0,0,0,0]]}, n=1)\n",
      "\n",
      "    #Find the closest neighbour of a single (sparse) query vector\n",
      "    table.search(vectors={'my_sparse_index':[{101:1,4578:1,102:1}]}, n=1)\n",
      "\n",
      "    #Find the 3 closest neighbours of 2 query vectors\n",
      "    table.search(vectors={'my_index': [[0,0,0,0,0,0,0,0,0,0], [1,1,1,1,1,1,1,1,1,1]]}, n=3)\n",
      "\n",
      "    # With aggregation and sorting\n",
      "    table.search(vectors={'my_index': [[0,0,0,0,0,0,0,0,0,0], [1,1,1,1,1,1,1,1,1,1]]},\n",
      "    n=3,\n",
      "    aggs={'sumSize': ['sum','size']},\n",
      "    group_by=['sym'],\n",
      "    sort_by=['sumSize'])\n",
      "\n",
      "    # Returns a subset of columns for each match\n",
      "    table.search(vectors={'my_index': [[0,0,0,0,0,0,0,0,0,0], [1,1,1,1,1,1,1,1,1,1]]}, n=3,\n",
      "                 aggs={'size': 'size'})\n",
      "    table.search(vectors={'my_index': [[0,0,0,0,0,0,0,0,0,0], [1,1,1,1,1,1,1,1,1,1]]}, n=3,\n",
      "                 aggs={'size': 'size', 'price': 'price'})\n",
      "\n",
      "    # Filter\n",
      "    table.search(vectors={'my_index': [[0,0,0,0,0,0,0,0,0,0], [1,1,1,1,1,1,1,1,1,1]]},\n",
      "    n=3,\n",
      "    filter=[['within','size',(5,999)],['like','sym','AAP*']])\n",
      "\n",
      "    # Customized distance name\n",
      "    table.search(vectors={'my_index': [[0,0,0,0,0,0,0,0,0,0], [1,1,1,1,1,1,1,1,1,1]]},\n",
      "    n=3,\n",
      "    options={\"distanceColumn\" 'myDist'})\n",
      "\n",
      "    # Index options\n",
      "    table.search(vectors=[[0,0,0,0,0,0,0,0,0,0], [1,1,1,1,1,1,1,1,1,1]], n=3, index_params=dict(efSearch=512))\n",
      "    table.search(vectors=[[0,0,0,0,0,0,0,0,0,0], [1,1,1,1,1,1,1,1,1,1]], n=3, index_params=dict(clusters=16))\n",
      "    ```\n",
      "\n",
      "Raises:\n",
      "    KDBAIException: Raised when an error occurs during search.\n",
      "\u001b[0;31mFile:\u001b[0m      ~/kx/similarity-search/kdbai-python-client/src/kdbai_client/api.py\n",
      "\u001b[0;31mType:\u001b[0m      method"
     ]
    }
   ],
   "source": [
    "?table.search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "bJgKFuFF1DqX",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "bJgKFuFF1DqX",
    "outputId": "ef1cf4eb-30e7-4dd4-aef2-7deaf4f4a6f8"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>__nn_distance</th>\n",
       "      <th>company_name</th>\n",
       "      <th>company_description</th>\n",
       "      <th>vectors</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.730767</td>\n",
       "      <td>Zoom</td>\n",
       "      <td>A video conferencing platform used for virtual...</td>\n",
       "      <td>[-0.047964468598365784, -0.001985185779631138,...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   __nn_distance company_name  \\\n",
       "0       0.730767         Zoom   \n",
       "\n",
       "                                 company_description  \\\n",
       "0  A video conferencing platform used for virtual...   \n",
       "\n",
       "                                             vectors  \n",
       "0  [-0.047964468598365784, -0.001985185779631138,...  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query = \"A company that helps facilitate meetings\"\n",
    "query_vector = list(embedding_model.embed([query]))[0].tolist()\n",
    "table.search(vectors={'vectorIndex': [query_vector]})[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9bb341f3",
   "metadata": {
    "id": "9bb341f3"
   },
   "source": [
    "<div class=\"alert alert-block alert-warning\">\n",
    "<b>Note:</b> The dimension of input query vectors must match the vector embedding dimensions in the table, defined in schema above.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49758e9d",
   "metadata": {
    "id": "49758e9d"
   },
   "source": [
    "<div class=\"alert alert-block alert-warning\">\n",
    "<b>Note:</b> The output was a list of length one, matching the number of vectors we input to the search. This can be indexed on position [0] to extract the dataframe corresponding to the single input vector.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0bc3820",
   "metadata": {
    "id": "f0bc3820"
   },
   "source": [
    "The closest matching neighbor for the query vector passed in is returned alongside the calculation of L2 ([Euclidean Distance](#https://en.wikipedia.org/wiki/Euclidean_distance)) similarity.\n",
    "\n",
    "We can also rerun the same query for more neighbors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "0c0cdda8",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 143
    },
    "id": "0c0cdda8",
    "outputId": "2bdb3554-562b-43a8-e461-d7e6f1266f3f"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>__nn_distance</th>\n",
       "      <th>company_name</th>\n",
       "      <th>company_description</th>\n",
       "      <th>vectors</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.730767</td>\n",
       "      <td>Zoom</td>\n",
       "      <td>A video conferencing platform used for virtual...</td>\n",
       "      <td>[-0.047964468598365784, -0.001985185779631138,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.714641</td>\n",
       "      <td>Booking.com</td>\n",
       "      <td>An online travel agency offering lodging reser...</td>\n",
       "      <td>[-0.011170933023095131, -0.01784929819405079, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.714121</td>\n",
       "      <td>Microsoft Teams</td>\n",
       "      <td>A collaboration platform for team communicatio...</td>\n",
       "      <td>[-0.04067962244153023, 0.02870851941406727, -0...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   __nn_distance     company_name  \\\n",
       "0       0.730767             Zoom   \n",
       "1       0.714641      Booking.com   \n",
       "2       0.714121  Microsoft Teams   \n",
       "\n",
       "                                 company_description  \\\n",
       "0  A video conferencing platform used for virtual...   \n",
       "1  An online travel agency offering lodging reser...   \n",
       "2  A collaboration platform for team communicatio...   \n",
       "\n",
       "                                             vectors  \n",
       "0  [-0.047964468598365784, -0.001985185779631138,...  \n",
       "1  [-0.011170933023095131, -0.01784929819405079, ...  \n",
       "2  [-0.04067962244153023, 0.02870851941406727, -0...  "
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find 3 closest neighbours of a single query vector\n",
    "table.search(vectors={'vectorIndex': [query_vector]}, n=3)[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0c9b0e8a",
   "metadata": {
    "id": "0c9b0e8a"
   },
   "source": [
    "And we can apply a filter to the search results. Here we use the '<>' filter, which keeps data that is not equal to a value."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "f1914948",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 143
    },
    "id": "f1914948",
    "outputId": "67995869-3114-4692-b34d-da8b6ad40bec"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>__nn_distance</th>\n",
       "      <th>company_name</th>\n",
       "      <th>company_description</th>\n",
       "      <th>vectors</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.730767</td>\n",
       "      <td>Zoom</td>\n",
       "      <td>A video conferencing platform used for virtual...</td>\n",
       "      <td>[-0.047964468598365784, -0.001985185779631138,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.714121</td>\n",
       "      <td>Microsoft Teams</td>\n",
       "      <td>A collaboration platform for team communicatio...</td>\n",
       "      <td>[-0.04067962244153023, 0.02870851941406727, -0...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.714121</td>\n",
       "      <td>Slack</td>\n",
       "      <td>A collaboration platform for team communicatio...</td>\n",
       "      <td>[-0.04067962244153023, 0.02870851941406727, -0...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   __nn_distance     company_name  \\\n",
       "0       0.730767             Zoom   \n",
       "1       0.714121  Microsoft Teams   \n",
       "2       0.714121            Slack   \n",
       "\n",
       "                                 company_description  \\\n",
       "0  A video conferencing platform used for virtual...   \n",
       "1  A collaboration platform for team communicatio...   \n",
       "2  A collaboration platform for team communicatio...   \n",
       "\n",
       "                                             vectors  \n",
       "0  [-0.047964468598365784, -0.001985185779631138,...  \n",
       "1  [-0.04067962244153023, 0.02870851941406727, -0...  \n",
       "2  [-0.04067962244153023, 0.02870851941406727, -0...  "
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find 3 closest neighbours of a single query vector\n",
    "table.search(\n",
    "    vectors={'vectorIndex': [query_vector]},\n",
    "    n=3,\n",
    "    filter=[(\"<>\", \"company_name\", \"Booking.com\")],\n",
    ")[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02c1b38d",
   "metadata": {
    "id": "02c1b38d"
   },
   "source": [
    "And also we can search passing more than one query vector."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "c6964035",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "c6964035",
    "outputId": "4824cfc4-cdcd-40c2-d5d8-0a33f45100d5"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[    Company Name\n",
       " 0        Spotify\n",
       " 1    Apple Music\n",
       " 2  YouTube Music,\n",
       "   Company Name\n",
       " 0     Facebook\n",
       " 1      Twitter\n",
       " 2    Instagram]"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "query1 = \"A company with a music-related product\"\n",
    "query2 = \"A social media company\"\n",
    "\n",
    "query1_vector = list(embedding_model.embed([query1]))[0].tolist()\n",
    "query2_vector = list(embedding_model.embed([query2]))[0].tolist()\n",
    "\n",
    "table.search(\n",
    "    vectors={'vectorIndex': [\n",
    "        query1_vector,\n",
    "        query2_vector,\n",
    "    ]},\n",
    "    n=3,\n",
    "    aggs={'Company Name': 'company_name'}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8aed9bc-72b2-4e70-b763-e7ce054557db",
   "metadata": {
    "id": "d8aed9bc-72b2-4e70-b763-e7ce054557db"
   },
   "source": [
    "## 6. Delete the KDB.AI Table\n",
    "\n",
    "We can use `table.drop()` to delete a table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "548a9d95-aac3-4d63-a87a-99eedfe55f07",
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "548a9d95-aac3-4d63-a87a-99eedfe55f07",
    "outputId": "e6b08842-86ef-4d63-ebf8-826142ee9538"
   },
   "outputs": [],
   "source": [
    "for t in session.database('default').tables:\n",
    "    if t.name == 'company_data':\n",
    "        t.drop()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8bc6d801-1371-48d0-98b4-0baa53bc8446",
   "metadata": {
    "id": "8bc6d801-1371-48d0-98b4-0baa53bc8446"
   },
   "source": [
    "<div class=\"alert alert-block alert-warning\">\n",
    "<b>Warning:</b> Once you drop a table, you cannot use it again.\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7672241-42b0-4798-90d7-95aa9fefe68c",
   "metadata": {
    "id": "a7672241-42b0-4798-90d7-95aa9fefe68c"
   },
   "source": [
    "## Next Steps\n",
    "\n",
    "Now that you’re successfully making indexes with KDB.AI, you can start inserting your own data or view more examples:\n",
    "- [PDF Document Search](../document_search)\n",
    "- [MRI Image Search](../image_search)\n",
    "- [Music Recommendation System](../music_recommendation)\n",
    "- [Sensor Pattern Matching](../pattern_matching)\n",
    "- [Retrieval Augmented Generation with LangChain](../retrieval_augmented_generation)\n",
    "- [Sentiment Analysis of Reviews](../sentiment_analysis)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  },
  "widgets": {
   "application/vnd.jupyter.widget-state+json": {
    "235128a18a4c47459047d47a32c61ebe": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "ProgressStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "ProgressStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "bar_color": null,
      "description_width": ""
     }
    },
    "30429cd24ff1429b9c4efafe0a95aff5": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "3dd7e95da10d4c9a987fb90ffccfd7ce": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "FloatProgressModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "FloatProgressModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "ProgressView",
      "bar_style": "success",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_30429cd24ff1429b9c4efafe0a95aff5",
      "max": 5,
      "min": 0,
      "orientation": "horizontal",
      "style": "IPY_MODEL_235128a18a4c47459047d47a32c61ebe",
      "value": 5
     }
    },
    "7223e3a082da40c0bfc607a2970d0acc": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_7e3bed2b82524fd083b76926897b9f18",
      "placeholder": "​",
      "style": "IPY_MODEL_77fd1cceeb0f4087909e08daa64a0f06",
      "value": "Fetching 5 files: 100%"
     }
    },
    "77fd1cceeb0f4087909e08daa64a0f06": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "7c590f17f5304259855eef5e727ef280": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HTMLModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HTMLModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HTMLView",
      "description": "",
      "description_tooltip": null,
      "layout": "IPY_MODEL_dd099525ffcb48bdbad60b52b8d1307f",
      "placeholder": "​",
      "style": "IPY_MODEL_93c2fb60dd1b4222bafa6df44af8264d",
      "value": " 5/5 [00:00&lt;00:00, 166.31it/s]"
     }
    },
    "7e3bed2b82524fd083b76926897b9f18": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "93c2fb60dd1b4222bafa6df44af8264d": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "DescriptionStyleModel",
     "state": {
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "DescriptionStyleModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "StyleView",
      "description_width": ""
     }
    },
    "c873cf1d86a642fbba9c5c3714ab279c": {
     "model_module": "@jupyter-widgets/controls",
     "model_module_version": "1.5.0",
     "model_name": "HBoxModel",
     "state": {
      "_dom_classes": [],
      "_model_module": "@jupyter-widgets/controls",
      "_model_module_version": "1.5.0",
      "_model_name": "HBoxModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/controls",
      "_view_module_version": "1.5.0",
      "_view_name": "HBoxView",
      "box_style": "",
      "children": [
       "IPY_MODEL_7223e3a082da40c0bfc607a2970d0acc",
       "IPY_MODEL_3dd7e95da10d4c9a987fb90ffccfd7ce",
       "IPY_MODEL_7c590f17f5304259855eef5e727ef280"
      ],
      "layout": "IPY_MODEL_d3ca822343ad490b9054cf08c913c48b"
     }
    },
    "d3ca822343ad490b9054cf08c913c48b": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    },
    "dd099525ffcb48bdbad60b52b8d1307f": {
     "model_module": "@jupyter-widgets/base",
     "model_module_version": "1.2.0",
     "model_name": "LayoutModel",
     "state": {
      "_model_module": "@jupyter-widgets/base",
      "_model_module_version": "1.2.0",
      "_model_name": "LayoutModel",
      "_view_count": null,
      "_view_module": "@jupyter-widgets/base",
      "_view_module_version": "1.2.0",
      "_view_name": "LayoutView",
      "align_content": null,
      "align_items": null,
      "align_self": null,
      "border": null,
      "bottom": null,
      "display": null,
      "flex": null,
      "flex_flow": null,
      "grid_area": null,
      "grid_auto_columns": null,
      "grid_auto_flow": null,
      "grid_auto_rows": null,
      "grid_column": null,
      "grid_gap": null,
      "grid_row": null,
      "grid_template_areas": null,
      "grid_template_columns": null,
      "grid_template_rows": null,
      "height": null,
      "justify_content": null,
      "justify_items": null,
      "left": null,
      "margin": null,
      "max_height": null,
      "max_width": null,
      "min_height": null,
      "min_width": null,
      "object_fit": null,
      "object_position": null,
      "order": null,
      "overflow": null,
      "overflow_x": null,
      "overflow_y": null,
      "padding": null,
      "right": null,
      "top": null,
      "visibility": null,
      "width": null
     }
    }
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}