diff --git a/basic-search/basic_search.ipynb b/basic-search/basic_search.ipynb new file mode 100644 index 0000000..c31cf20 --- /dev/null +++ b/basic-search/basic_search.ipynb @@ -0,0 +1,427 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "c60c757f-2f3a-4a84-9a8f-09adfc1d3d72", + "metadata": {}, + "source": [ + "## Overview\n", + "\n", + "AVS provides a gRPC API and a Python client that developers can use to create AI applications leveraging its search capability. This example shows basic usage of the Aerospike Vector Search Python client including, writing, reading, and searching for vectors with AVS. For more detailed client documentation visit the [Python client's read the docs page](https://aerospike-vector-search-python-client.readthedocs.io/en/latest/).\n", + "\n", + "### Prerequisites\n", + "\n", + "- Python 3.9 or higher\n", + "- pip 9.0.1 or higher\n", + "- An accessible AVS deployment\n", + "\n", + "## Installing AVS\n", + "Before getting started, you need to setup a running Aerospike Vector Search (AVS) deployment.\n", + "See the [AVS installation guide](https://aerospike.com/docs/vector/install) for tips on setting up AVS.\n", + "When finished, store your AVS instance's IP address and port to use later in this demo:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "246d7e0c-57ea-433f-8e5e-a1d72f83e35b", + "metadata": {}, + "outputs": [], + "source": [ + "AVS_HOST = \"\"\n", + "AVS_PORT = 5000" + ] + }, + { + "cell_type": "markdown", + "id": "34efa0ea-abfd-46a8-92d7-fbfd2dcf8139", + "metadata": {}, + "source": [ + "## Installing the Aerospike Vector Search Client\n", + "\n", + "The Aerospike Vector Search Python client is distributed on pypi so you can install directly with pip." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "2b5c38d3-2d9d-4565-8a56-d69ca02bb008", + "metadata": {}, + "outputs": [], + "source": [ + "!pip install aerospike-vector-search" + ] + }, + { + "cell_type": "markdown", + "id": "bb76a7c9-0428-43c7-84c5-e36c727f338e", + "metadata": {}, + "source": [ + "## Importing Python clients\n", + "\n", + "The aerospike-vector-search package provides two separate clients:\n", + "\n", + "* **avs_client** performs database operations with vector data. The client supports Hierarchical Navigable Small World (HNSW) vector searches, allowing users to find vectors similar to a given query vector within an index.\n", + "\n", + "* **avs_admin_client** conducts AVS administrative operations, such as creating indexes, querying index information, and dropping indexes.\n", + "\n", + "The package also provides a `types` module that contains classes necessary for interacting with the various client APIs." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "13d72e68-1fac-4655-9e4f-f89d9f95d71c", + "metadata": {}, + "outputs": [], + "source": [ + "from aerospike_vector_search import types\n", + "from aerospike_vector_search import AdminClient, Client" + ] + }, + { + "cell_type": "markdown", + "id": "27700922-0438-4eac-80f3-288b4db81c12", + "metadata": {}, + "source": [ + "## Creating a Vector Admin Client\n", + "\n", + "Initialize a new admin client by providing one or more seed hosts to which the client can connect." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c8a4330b-1ea4-4433-ae61-3b89edbf0a5c", + "metadata": {}, + "outputs": [], + "source": [ + "# Admin client configuration\n", + "# LISTENER_NAME corresponds to the AVS advertised_listener config.\n", + "# https://aerospike.com/docs/vector/operate/configuration#advertised-listener\n", + "# this is often needed when connection to AVS clusters in the cloud\n", + "LISTENER_NAME = None\n", + "# LOAD_BALANCED is True if the AVS cluster is load balanced\n", + "# using a load balancer with AVS is best practice and even works \n", + "# with a single node AVS cluster that is not load balanced\n", + "LOAD_BALANCED = True\n", + "\n", + "admin_client = AdminClient(\n", + " seeds=types.HostPort(host=AVS_HOST, port=AVS_PORT),\n", + " listener_name=LISTENER_NAME,\n", + " is_loadbalancer=LOAD_BALANCED,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "7479d904-22b4-49d7-878d-357c20b7478c", + "metadata": {}, + "source": [ + "## Create an index\n", + "\n", + "To search across a set of vectors, you need to create an index associated with those vectors. AVS uses an index to traverse the HNSW neighborhoods to perform queries.\n", + "\n", + "See [Manage AVS indexes](https://aerospike.com/docs/vector/operate/index-management) for details about creating an index." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "ee1a4813-48fe-4eed-9a7b-7078fd3c530d", + "metadata": {}, + "outputs": [], + "source": [ + "# Index creation arguments\n", + "# NAMESPACE is the namespace that the indexed data will be stored in\n", + "NAMESPACE = \"test\"\n", + "# INDEX_NAME is the name of the HNSW index to create\n", + "INDEX_NAME = \"basic_index\"\n", + "# VECTOR_FIELD is the Aerospike record bin that stores its vector data\n", + "# The created index will use the data in this bin to perform nearest neighbor searches etc\n", + "VECTOR_FIELD = \"vector\"\n", + "# DIMENSIONS is the dimensionality of the vectors\n", + "DIMENSIONS = 2\n", + "\n", + "try:\n", + " print(\"creating index\")\n", + " admin_client.index_create(\n", + " namespace=NAMESPACE,\n", + " name=INDEX_NAME,\n", + " vector_field=VECTOR_FIELD,\n", + " dimensions=DIMENSIONS,\n", + " )\n", + "except Exception as e:\n", + " print(\"failed creating index \" + str(e))\n", + " pass" + ] + }, + { + "cell_type": "markdown", + "id": "e6f632d9-799a-4a36-b353-fc44ee66f166", + "metadata": {}, + "source": [ + "## Create a Vector Client.\n", + "\n", + "Initialize a new client by providing one or more seed hosts to which the client can connect." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "e6d2e8f8-3dee-4e5a-9d7b-86f5d5291199", + "metadata": {}, + "outputs": [], + "source": [ + "client = Client(\n", + " seeds=types.HostPort(host=AVS_HOST, port=AVS_PORT),\n", + " listener_name=LISTENER_NAME,\n", + " is_loadbalancer=LOAD_BALANCED,\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "2fe44b27-50ed-4658-8269-9a39e7908c26", + "metadata": {}, + "source": [ + "## Add Vector Entries\n", + "Vectors records must exist in AVS before searches can be performed.\n", + "\n", + "To insert records, use the `upsert` method and specify the following values when writing a record:\n", + "\n", + "* `namespace` - Namespace in which the index exists.\n", + "* `setName` (optional) - Set in which to place the record.\n", + "* `key` - Primary identifier for your record.\n", + "* `record data` - Map of any data you want to associate with your vector.\n", + "\n", + "The following code inserts ten vector records: " + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "8bca1b78-06e3-4447-8513-9ef9f4813904", + "metadata": {}, + "outputs": [], + "source": [ + "# set_name is the Aerospike set to write the records to\n", + "SET_NAME = \"basic-set\"\n", + "\n", + "print(\"inserting vectors\")\n", + "for i in range(10):\n", + " key = \"r\" + str(i)\n", + " client.upsert(\n", + " namespace=NAMESPACE,\n", + " set_name=SET_NAME,\n", + " key=key,\n", + " record_data={\n", + " \"url\": f\"http://host.com/data{i}\",\n", + " \"vector\": [i * 1.0, i * 1.0],\n", + " \"map\": {\"a\": \"A\", \"inlist\": [1, 2, 3]},\n", + " \"list\": [\"a\", 1, \"c\", {\"a\": \"A\"}],\n", + " },\n", + " )" + ] + }, + { + "cell_type": "markdown", + "id": "5af9d461-818c-40b7-9a39-ec329b93f170", + "metadata": {}, + "source": [ + "## Wait for Index Construction\n", + "\n", + "After inserting vectors into AVS, it will take some time to build the index. If the index is not complete, vector search results may be inaccurate. If you want confirmation that index construction is complete, you can do the following:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "c78d3f25-5e45-4e78-980f-25fd847dbb38", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"waiting for indexing to complete\")\n", + "client.wait_for_index_completion(namespace=NAMESPACE, name=INDEX_NAME)" + ] + }, + { + "cell_type": "markdown", + "id": "a8b19f64-7b12-45fe-b68c-faaf9e342dbb", + "metadata": {}, + "source": [ + "## Check if a Vector is Indexed\n", + "\n", + "Alternatively, you can check individual records to see if they have completed indexing." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "53cd2051-2ce2-43f6-847b-a710438e11d8", + "metadata": {}, + "outputs": [], + "source": [ + "status = client.is_indexed(\n", + " namespace=NAMESPACE,\n", + " set_name=SET_NAME,\n", + " key=key,\n", + " index_name=INDEX_NAME,\n", + ")\n", + "\n", + "print(\"indexed: \", status)" + ] + }, + { + "cell_type": "markdown", + "id": "b22c1b32-7b04-44a4-adec-c77a2c973df1", + "metadata": {}, + "source": [ + "## Searching\n", + "\n", + "After vectors have been indexed, you can begin searching them by providing a vector for search. This generally entails running your machine learning model on user input, and then performing a search using the generated embedding." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "a8feaa88-1058-4c6b-a095-8a65bcce7d15", + "metadata": {}, + "outputs": [], + "source": [ + "print(\"querying\")\n", + "for i in range(10):\n", + " print(\" query \" + str(i))\n", + " results = client.vector_search(\n", + " namespace=NAMESPACE,\n", + " index_name=INDEX_NAME,\n", + " query=[i * 1.0, i * 1.0],\n", + " limit=3,\n", + " )\n", + " for result in results:\n", + " print(str(result.key.key) + \" -> \" + str(result.fields))" + ] + }, + { + "cell_type": "markdown", + "id": "8f7cc4bb-c4c4-4cd1-888b-b1c90cd93650", + "metadata": {}, + "source": [ + "## Retrieving Vector Data\n", + "\n", + "You can read records from AVS with the AVS client `get` method." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "b86194a3-d20d-4395-b6cd-30cdc0a2c274", + "metadata": {}, + "outputs": [], + "source": [ + "key = \"r0\"\n", + "\n", + "result = client.get(\n", + " namespace=NAMESPACE,\n", + " key=key,\n", + " set_name=SET_NAME,\n", + ")\n", + "\n", + "print(str(result.key.key) + \" -> \" + str(result.fields))" + ] + }, + { + "cell_type": "markdown", + "id": "3ea9c937-c719-4801-82c5-2601345c49f9", + "metadata": {}, + "source": [ + "## AVS Client using Asyncio\n", + "\n", + "The `aerospike-vector-search` module provides an [aio module](https://aerospike-vector-search-python-client.readthedocs.io/en/latest/aio.html) with asynchronous clients that replace any client methods with coroutine methods. The asynchronous client is initialized in the same way as the synchronous clients. Simply add `await` in front of synchronous code to convert code examples:" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "cd8baedb-f8b2-468c-90f5-5a8a32ee65f4", + "metadata": {}, + "outputs": [], + "source": [ + "from aerospike_vector_search.aio import Client as asyncClient\n", + "\n", + "async_client = asyncClient(\n", + " seeds=types.HostPort(host=AVS_HOST, port=AVS_PORT),\n", + " listener_name=LISTENER_NAME,\n", + " is_loadbalancer=LOAD_BALANCED,\n", + ")\n", + "\n", + "# Use await on client methods to await completion of the coroutine\n", + "results = await async_client.vector_search(\n", + " namespace=NAMESPACE,\n", + " index_name=INDEX_NAME,\n", + " query=[8.0, 8.0],\n", + " limit=3,\n", + ")\n", + "\n", + "for result in results:\n", + " print(str(result.key.key) + \" -> \" + str(result.fields))" + ] + }, + { + "cell_type": "markdown", + "id": "3956f261-0b17-4478-8f2d-470d6a6b32aa", + "metadata": {}, + "source": [ + "## Cleanup\n", + "\n", + "Make sure to close the AVS clients when you are done using them." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "f411d964-3524-450a-be0e-a3e02421c398", + "metadata": {}, + "outputs": [], + "source": [ + "admin_client.close()\n", + "client.close()\n", + "async_client.close()" + ] + }, + { + "cell_type": "markdown", + "id": "9f9f2511-6f50-456f-b00c-e590acdc548e", + "metadata": {}, + "source": [ + "## Next Steps\n", + "\n", + "See the following pages for more examples and AVS Python client information.\n", + "- [AVS Python Langchain example](https://colab.research.google.com/github/aerospike/langchain/blob/master/docs/docs/integrations/vectorstores/aerospike.ipynb?authuser=1)\n", + "- [AVS Python client documentation](https://aerospike-vector-search-python-client.readthedocs.io/en/latest/index.html)\n", + "- [AVS Examples on Github](https://github.com/aerospike/aerospike-vector-search-examples)" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.2" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +}