feat: Add search/cloud-function/python (GoogleCloudPlatform#861)

This is a long awaited update of a now closed PR GoogleCloudPlatform#314. _(blast from the past)_

Ready:
- This should be very easy to deploy and use locally, based on Cloud
Functions as a standardization
- There is minimal response simplification based on the search
configuration, which should help people get started and
- There are unit tests which mock the APIs
- There are integration tests which do very basic functionality
validation, calling the API (more in the future)

Future:
- Right now this is python only, but I intend to make the same
abstraction for a few other languages (polyglot FTW)
- Creation of private integration tests based on this framework
validating combinations of features
  - Summarization + Unstructured data + Chunk mode
  - Summarization + Structured data
  - Summarization + Blended data
- There's no UX today but we could consider adding one
- There's a placeholder config to do Generation based on Vertex Search
retrieval via GEMINI or GENERATE_GROUNDED_ANSWERS, which might be added
to the python client before creating variants (ref
go/vertex-ai-search-retrieval-gemini-generate)

---------

Co-authored-by: Holt Skinner <[email protected]>
Co-authored-by: Mend Renovate <[email protected]>
Co-authored-by: Alan Blount <[email protected]>
Co-authored-by: Holt Skinner <[email protected]>
Co-authored-by: Owl Bot <gcf-owl-bot[bot]@users.noreply.github.com>

.github/actions/spelling/allow.txt

@@ -70,7 +70,9 @@ Gisting
 Glickman
 Googlers
 HIDPI
+HMO
 HREDRAW
+HSA
 Hamers
 Hickson
 Hmmm
@@ -342,6 +344,7 @@ mec
 meme
 memes
 metadatas
+miranda
 mpn
 nbconvert
 nbfmt

search/cloud-function/README.md

@@ -0,0 +1,128 @@
+# Vertex AI Search accessed via Google Cloud Functions
+
+This directory contains several versions of approximately the same
+implementation.
+
+The functions can be deployed to
+[Cloud functions](https://cloud.google.com/functions/) and can be modified to
+supports many different triggers and use cases. Each can also
+[be deployed locally](https://cloud.google.com/functions/docs/running/overview)
+which allows easy experimentation and iteration.
+
+This example is powered by
+[Vertex AI Search](https://cloud.google.com/generative-ai-app-builder/docs/enterprise-search-introduction)
+which does many different things, including **Document & Intranet Search**,
+**Recommendations** and **Grounding and RAG** out-of-the-box (For more
+information, see the blog post
+[Your RAG powered by Google Search](https://cloud.google.com/blog/products/ai-machine-learning/rags-powered-by-google-search-technology-part-1)).
+
+If you want even more control see
+[Vertex AI Search Component APIs](https://cloud.google.com/generative-ai-app-builder/docs/builder-apis),
+but first explore the out-of-the-box offering because it's easy to setup.
+
+This example is for the out-of-the-box Vertex AI Search supporting many
+configurations and data types.
+
+## Pre-requisites
+
+Before you can use these functions to query Vertex AI Search, you need to create
+and populate a search "data store"; read through instructions in
+[get started with generic search](https://cloud.google.com/generative-ai-app-builder/docs/try-enterprise-search).
+These functions could easily be adapted to other types of Vertex AI Search like
+[generic recommendations](https://cloud.google.com/generative-ai-app-builder/docs/try-generic-recommendations),
+[media search](https://cloud.google.com/generative-ai-app-builder/docs/try-media-search),
+[media recommendations](https://cloud.google.com/generative-ai-app-builder/docs/try-media-recommendations),
+[healthcare search](https://cloud.google.com/generative-ai-app-builder/docs/create-data-store-hc),
+or even
+[retail product discovery](https://cloud.google.com/solutions/retail-product-discovery#documentation).
+
+You'll need to collect the following details from your search app data store:
+
+```python
+PROJECT_ID = "YOUR_PROJECT_ID"  # alphanumeric
+LOCATION = "global"  # or an alternate location
+DATA_STORE_ID = "YOUR_DATA_STORE_ID"  # not the app id, alphanumeric
+```
+
+Additionally you'll need to keep track of some of the choices you make when you
+configure Vertex AI Search.
+
+### Type of data source
+
+<!-- textlint-disable -->
+
+- UNSTRUCTURED
+- STRUCTURED
+- WEBSITE
+- BLENDED
+
+<!-- textlint-enable -->
+
+```python
+ENGINE_DATA_TYPE = UNSTRUCTURED
+```
+
+### Type of chunks to return
+
+- DOCUMENT_WITH_SNIPPETS
+- DOCUMENT_WITH_EXTRACTIVE_SEGMENTS
+- CHUNK
+- NONE
+
+```python
+ENGINE_CHUNK_TYPE = DOCUMENT_WITH_EXTRACTIVE_SEGMENTS
+```
+
+### Type of summarization
+
+- NONE results only
+- VERTEX_AI_SEARCH LLM add on provided by
+[Vertex AI Search](https://cloud.google.com/generative-ai-app-builder/docs/enterprise-search-introduction)
+<!-- NOT ready yet
+- GENERATE_GROUNDED_ANSWERS use the
+  [Generate grounded answers with RAG](https://cloud.google.com/generative-ai-app-builder/docs/grounded-gen)
+  provided by
+  [Vertex AI Search Builder APIs](https://cloud.google.com/generative-ai-app-builder/docs/builder-apis)
+- GEMINI use one of the Gemini models to generate an answer from the results -->
+
+```python
+SUMMARY_TYPE = VERTEX_AI_SEARCH
+```
+
+## Architecture
+
+1. Vertex AI Search is an API hosted on Google Cloud
+2. You will call that API via a Google Cloud Function, which exposes its own API
+3. Your users will the Google Cloud Function API, via your custom app or UI
+
+```mermaid
+flowchart LR
+    A[fa:fa-search Vertex Search AI] --> B(Google Cloud Function)
+    B --> C[My App Server]
+    C -->|One| D[fa:fa-laptop web]
+    C -->|Two| E[fa:fa-mobile mobile]
+```
+
+## Use case: RAG / Grounding
+
+Any time you have more source data than can fit into a LLM context window, you
+could benefit from RAG (Retrieval Augmented Generation). The more data you have,
+the more important search is - to get the relevant chunks into the prompt of the
+LLM.
+
+- **Retrieve** relevant search results, with text chunks (snippets or segments)
+- **Augmented Generation** uses Gemini to generate an answer or summary grounded
+  on the relevant search results
+
+## Use case: Agent Tool (Knowledge Base)
+
+A natural extension of RAG / Grounding is agentic behavior.
+
+Whether creating a basic chatbot or a sophisticated tool using multi-agent
+system, you're always going to need search based RAG. The better the search
+quality the better the agent response based on your source data.
+
+For more on agents, check out
+[Vertex AI Agent Builder Use Cases](https://cloud.google.com/products/agent-builder?hl=en#common-uses)
+and
+[https://github.com/GoogleCloudPlatform/generative-ai](https://github.com/GoogleCloudPlatform/generative-ai).

search/cloud-function/python/.env.example

@@ -0,0 +1,6 @@
+DATA_STORE_ID=your-data-store-id-here
+ENGINE_CHUNK_TYPE=DOCUMENT_WITH_EXTRACTIVE_SEGMENTS
+ENGINE_DATA_TYPE=UNSTRUCTURED
+LOCATION=global
+PROJECT_ID=your-project-id-here
+SUMMARY_TYPE=VERTEX_AI_SEARCH

search/cloud-function/python/README.md

@@ -0,0 +1,127 @@
+# Vertex AI Search accessed via Google Cloud Functions
+
+This example is based on the
+[Python client for the Vertex AI Search API](https://cloud.google.com/generative-ai-app-builder/docs/libraries#client-libraries-usage-python),
+which will get search results, snippets, metadata, and the LLM summary grounded
+on search results. This is implemented in the `vertex_ai_search_client.py` file.
+
+That functionality is exposed on a REST API which is implemented in `main.py`
+intended to be deployed to a Google Cloud Function using an HTTPS trigger on a
+Python 3 runtime;
+[read more here](https://cloud.google.com/functions/docs/samples/functions-http-content#functions_http_content-python).
+
+**[Read more about Vertex AI Search accessed via Google Cloud Functions](../)**
+
+## Environment Variables
+
+The following environment variables are required for both local development and
+deployment:
+
+- `PROJECT_ID`: Your Google Cloud project ID
+- `LOCATION`: The location of your Vertex AI Search data store
+- `DATA_STORE_ID`: The ID of your Vertex AI Search data store
+- `ENGINE_DATA_TYPE`: Type of data in the engine (0-3)
+- `ENGINE_CHUNK_TYPE`: Type of chunking used (0-3)
+- `SUMMARY_TYPE`: Type of summary used (0-3)
+
+## Local Development
+
+### Setup
+
+1. Ensure you have the Google Cloud SDK installed and configured.
+2. Clone this repository and navigate to the project directory.
+3. Set up your environment variables:
+
+```bash
+gcloud auth login
+bash setup_env.sh
+```
+
+Alternatively, you can manually create and edit a `.env` file with the required
+variables.
+
+### Run locally
+
+Run this code locally via **Functions Framework** or **Functions Emulator**;
+[read more about running cloud functions locally](https://cloud.google.com/functions/docs/running/overview).
+
+```bash
+pip install -r requirements.txt
+pip install functions-framework
+functions-framework --target=vertex_ai_search
+```
+
+In a different terminal, execute a `POST` search query based on your data:
+
+```bash
+export SEARCH_TERM="What is the ... for ...?"
+curl -m 310 -X POST localhost:8080 \
+-H "Content-Type: application/json" \
+-d "{\"search_term\": \"${SEARCH_TERM}\"}"
+```
+
+### Run tests
+
+#### Unit tests
+
+These tests mock the API interactions and should run quickly:
+
+```bash
+pip install pytest
+pytest test_vertex_ai_search_client.py
+```
+
+#### Integration tests
+
+These tests actually call the Vertex AI Search API and depend on your data
+stores being configured in Vertex AI Search:
+
+```bash
+pip install pytest
+pytest test_integration_vertex_ai_search_client.py
+```
+
+## Deployment
+
+To deploy this function to Google Cloud:
+
+1. Ensure you have set up the required environment variables (see Environment
+   Variables section).
+2. Run the following command:
+
+```bash
+gcloud functions deploy vertex_ai_search --runtime python39 --trigger-http --allow-unauthenticated
+```
+
+You will get back a URL for triggering the function.
+
+## Usage
+
+After deployment, you can use the function as follows:
+
+```bash
+curl -X POST https://YOUR_FUNCTION_URL \
+-H "Content-Type: application/json" \
+-d '{"search_term": "your search query"}'
+```
+
+Replace `YOUR_FUNCTION_URL` with the URL of your deployed function, and fill in
+the search query.
+
+If you run into problems, go to
+[Google Cloud Functions](https://console.cloud.google.com/functions), find the
+function you just deployed, and review the logs for informative errors. Perhaps
+you need to setup
+[Google Cloud IAM](https://cloud.google.com/functions/docs/reference/iam) roles
+or permissions.
+
+## Customization
+
+This implementation provides a basic way to access and control your queries to
+the Vertex AI Search API. It simplifies CORS and bearer token authentication,
+and allows for some minor customization of inputs and outputs.
+
+If you require more extensive customization, consider using an orchestration
+framework like [LangChain](https://www.langchain.com/) or
+[LlamaIndex](https://www.llamaindex.ai/) which have Vertex AI Search
+integrations.