AWS OpenSearch with OpenSearchVectorSearch generate "failed to create query: Field 'vector_field' is not knn_vector type."

[sebastianherreramonterrosa]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

from langchain.vectorstores import OpenSearchVectorSearch
from langchain.embeddings import BedrockEmbeddings
from requests_aws4auth import AWS4Auth
from opensearchpy import OpenSearch, RequestsHttpConnection
import boto3

service = "aoss"
credentials = boto3.Session().get_credentials()
region = boto3.Session().region_name
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

bedrock_client = boto3.client(service_name="bedrock-runtime")

embeddings = BedrockEmbeddings(
    client=bedrock_client,
    model_id="amazon.titan-embed-text-v1"  # Ajusta según el modelo que estés usando
)

opensearch_domain_endpoint = ...
opensearch_index = ...
vector_field = ...

opensearch = OpenSearchVectorSearch(
    index_name=opensearch_index,
    embedding_function=embeddings,
    opensearch_url=opensearch_domain_endpoint,
    http_auth=awsauth,
    connection_class=RequestsHttpConnection,
    use_ssl=True,
    verify_certs=True,
    vector_field=vector_field,
)

query = "Tu consulta aquí"
results = opensearch.similarity_search(
    query,
    k=5,
)

Description

I have created a knowledge base in aws bedrock. I created the knowledge base with default configuration. A vector database has been successfully created in aws OpenSearchService. I want to access that vector database with langchain. When I try I get this error

RequestError: RequestError(400, 'search_phase_execution_exception', "failed to create query: Field 'vector_field' is not knn_vector type."

System Info

langchain==0.2.5
python 3.11

[dosubot[bot]]

To resolve the error "RequestError: RequestError(400, 'search_phase_execution_exception', 'failed to create query: Field 'vector_field' is not knn_vector type.'" and successfully query your vector database in AWS OpenSearchService using LangChain, you need to ensure that the field vector_field in your Elasticsearch index is of the correct type, which should be knn_vector.

Here is how you can define the index mapping correctly:

1. Ensure that the index is created with the correct mapping for the vector field.
2. Use the OpenSearchVectorSearch class to create the index with the appropriate settings.

Here is an example of how to create the index with the correct mapping:

from langchain.vectorstores import OpenSearchVectorSearch
from langchain.embeddings import BedrockEmbeddings
from requests_aws4auth import AWS4Auth
from opensearchpy import OpenSearch, RequestsHttpConnection
import boto3

service = "aoss"
credentials = boto3.Session().get_credentials()
region = boto3.Session().region_name
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

bedrock_client = boto3.client(service_name="bedrock-runtime")

embeddings = BedrockEmbeddings(
    client=bedrock_client,
    model_id="amazon.titan-embed-text-v1"  # Adjust according to the model you are using
)

opensearch_domain_endpoint = "your-opensearch-domain-endpoint"
opensearch_index = "your-index-name"
vector_field = "vector_field"

opensearch = OpenSearchVectorSearch(
    index_name=opensearch_index,
    embedding_function=embeddings,
    opensearch_url=opensearch_domain_endpoint,
    http_auth=awsauth,
    connection_class=RequestsHttpConnection,
    use_ssl=True,
    verify_certs=True,
    vector_field=vector_field,
    dim=128,  # Specify the dimensions of your vectors
)

# Define the index mapping
index_mapping = {
    "settings": {"index": {"knn": True, "knn.algo_param.ef_search": 512}},
    "mappings": {
        "properties": {
            "vector_field": {
                "type": "knn_vector",
                "dimension": 128,
                "method": {
                    "name": "hnsw",
                    "space_type": "l2",
                    "engine": "nmslib",
                    "parameters": {"ef_construction": 512, "m": 16},
                },
            }
        }
    },
}

# Create the index if it does not exist
if not opensearch.client.indices.exists(index=opensearch_index):
    opensearch.client.indices.create(index=opensearch_index, body=index_mapping)

# Now you can add documents and perform queries
query = "Your query here"
results = opensearch.similarity_search(query, k=5)

Ensure that the vector_field is set to knn_vector and the dimension parameter matches the dimensions of your vectors. This should resolve the error and allow you to query your vector database successfully ^[1][2].

To continue talking to Dosu, mention @dosu.

[sebastianherreramonterrosa]

I did what you told me and it still doesn't work

[dosubot[bot]]

If the issue persists, please ensure the following steps are correctly implemented:

1. Index Creation: Verify that the index is created with the correct mapping for the knn_vector field. The field must be of type knn_vector and have the correct dimensions.

2. Index Mapping: Ensure the index mapping is correctly applied. Here is a more detailed example:

from opensearchpy import OpenSearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth
import boto3

# AWS credentials and region
service = "aoss"
credentials = boto3.Session().get_credentials()
region = boto3.Session().region_name
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

# OpenSearch client
opensearch = OpenSearch(
    hosts=[{'host': 'your-opensearch-domain-endpoint', 'port': 443}],
    http_auth=awsauth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection
)

# Index name and vector field
index_name = "your-index-name"
vector_field = "vector_field"
dim = 128  # Specify the dimensions of your vectors

# Define the index mapping
index_mapping = {
    "settings": {"index": {"knn": True, "knn.algo_param.ef_search": 512}},
    "mappings": {
        "properties": {
            vector_field: {
                "type": "knn_vector",
                "dimension": dim,
                "method": {
                    "name": "hnsw",
                    "space_type": "l2",
                    "engine": "nmslib",
                    "parameters": {"ef_construction": 512, "m": 16},
                },
            }
        }
    },
}

# Create the index if it does not exist
if not opensearch.indices.exists(index=index_name):
    opensearch.indices.create(index=index_name, body=index_mapping)

# Verify the mapping
mapping = opensearch.indices.get_mapping(index=index_name)
print(mapping)

3. Document Insertion: Ensure that documents are inserted into the index with the correct vector field.

4. Query Execution: Ensure the query is correctly formatted and targets the knn_vector field.

If the issue still persists, please provide the exact error message and any relevant code snippets. This will help in diagnosing the problem more accurately.

To continue talking to Dosu, mention @dosu.