Flexible customization of the Document Class to set page_content and metadata properties on vector store search results

[paychex-ssmithrand]

Checked

• I searched existing ideas and did not find a similar one
• I added a very descriptive title
• I've clearly described the feature request and motivation for it

Feature request

Currently, one set of environment variables define what fields should be set as page_content and metadata on a retrieved Document from Azure AI Search.

It would be nice to specify what returned fields from the index define these properties on the creation of the Azure Search class. Mainly helpful in multi-index retrieval with different field configurations. There is already a fields list that can be passed in to override the environment variables for search behavior.

Motivation

Stronger support for multi-index retrieval with non-conforming configurations on them.

Proposal (If applicable)

Something like:

AzureSearch(
      azure_search_endpoint=os.getenv("AZURE_SEARCH_ENDPOINT"),
      azure_search_key=os.getenv("AZURE_SEARCH_API_KEY"),
      index_name="my-index",
      semantic_configuration_name="my-semantic-config",
      embedding_function=embeddings.embed_query,
      fields = [
          SimpleField(
            name="id",
            type=SearchFieldDataType.String,
            key=True,
            filterable=True,
          ),
          SearchableField(
            name="content_chunk",
            type=SearchFieldDataType.String,
            searchable=True,
            page_content=True #new param on SearchableField class
          ),
          SearchField(
            name="content_vector",
            type=SearchFieldDataType.Collection(SearchFieldDataType.Single),
            searchable=True,
            vector_search_dimensions=1536,
            vector_search_configuration="default",
          ),
          SearchableField(
            name="metadata",
            type=SearchFieldDataType.String,
          ),
      ]
  )

[paychex-ssmithrand]

Bump. I'd really like to get some eyes on this.

[eyurtsev]

@paychex-ssmithrand This make sense. The existing implementation is relying on env variables, which generally should not be used for flow control. This is a community maintained implementation, so if you're up for it you can try to implement the logic that's needed to improve the vectorstore. Alternatively, you can also copy and paste the implementation into your code base directly and adapt it until it works for your use case (and maybe contribute a far better version :)).