Add GET api/v1/indexes to Lambda searcher

[tyrwzl]

Description

Closes #5039.
Add GET /api/v1/indexes endpoint to Lambda Searcher to use quickwit Lambda as Grafana datasource.

How was this PR tested?

1. Locally running Lambda searcher by using cargo lambda.

export AWS_ACCESS_KEY_ID="****"
export AWS_SECRET_ACCESS_KEY="****"
export AWS_SESSION_TOKEN="****"
export AWS_REGION="us-east-1"
export AWS_LAMBDA_HTTP_IGNORE_STAGE_IN_PATH="true"
export QW_LAMBDA_INDEX_BUCKET="quickwitlambdastack-quickwitindexstore****"
export QW_LAMBDA_INDEX_ID="hdfs-logs"
export QW_LAMBDA_METASTORE_BUCKET="quickwitlambdastack-quickwitindexstore****"
export RUST_LOG="quickwit=info"
cargo lambda watch

2. Setup Grafana (https://quickwit.io/docs/get-started/tutorials/trace-analytics-with-grafana)
   • url: http://host.docker.internal:9000/lambda-url/searcher/api/v1
   • Index ID: hdfs-logs

3. Explorer quickwit data
   

[CLAassistant]

All committers have signed the CLA.

[rdettai]

Thanks for this contribution and for testing it on Grafana! It's amazing to see it works!

Nit: have you tried running the CDK examples already? If so, you could add this new endpoint to the tests here

quickwit/distribution/lambda/cdk/cli.py

Lines 420 to 462 in f8590d5

	def test_mock_data_endpoints():
	apigw_url = _get_cloudformation_output_value(
	app.MOCK_DATA_STACK_NAME, mock_data_stack.API_GATEWAY_EXPORT_NAME
	)
	def req(method, path, body=None, expected_status=200):
	conn = http.client.HTTPSConnection(urlparse(apigw_url).netloc)
	conn.request(
	method,
	path,
	body,
	headers={"x-api-key": os.getenv("SEARCHER_API_KEY")},
	)
	response = conn.getresponse()
	print(f"{method} {path}")
	headers = {k: v for (k, v) in response.getheaders()}
	body = _decompress_if_gzip(response.read(), headers)
	if response.status != expected_status:
	print(f"[{response.status}] => {body}")
	exit(1)
	else:
	print(f"[{response.status}] => {json.dumps(json.loads(body))[0:100]}")
	req("GET", f"/api/v1/{mock_sales_index_id}/search?query=animal")
	req(
	"POST",
	f"/api/v1/{mock_sales_index_id}/search",
	'{"query":"quantity:>5", "max_hits": 10}',
	)
	req("GET", f"/api/v1/_elastic/{mock_sales_index_id}/_search?q=animal")
	req(
	"POST",
	f"/api/v1/_elastic/{mock_sales_index_id}/_search",
	'{"query":{"bool":{"must":[{"range":{"quantity":{"gt":5}}}]}},"size":10}',
	)
	req("GET", f"/api/v1/_elastic/{mock_sales_index_id}/_field_caps?fields=quantity")
	# expected errors
	req(
	"GET",
	f"/api/v1/_elastic/{mock_sales_index_id}/_search?query=animal",
	expected_status=400,
	)
	req("GET", f"/api/v1/_elastic/_search?q=animal", expected_status=501)

. These are some sanity checks on the end to end CDK deployment. You can run them by first deploying the stack:

export CDK_ACCOUNT=123456789
export CDK_REGION=us-east-1
make bootstrap
make deploy-mock-data
make test-mock-data-endpoints

[tyrwzl]

Thank you for instructions.
Added GET api/v1/indexes test.

GET /api/v1/mock-sales/search?query=animal
[200] => {"num_hits": 0, "hits": [], "elapsed_time_micros": 51301, "errors": []}
POST /api/v1/mock-sales/search
[200] => {"num_hits": 0, "hits": [], "elapsed_time_micros": 1600, "errors": []}
GET /api/v1/_elastic/mock-sales/_search?q=animal
[200] => {"took": 1, "timed_out": false, "_shards": {"total": 1, "successful": 1, "skipped": 0, "failed": 0},
POST /api/v1/_elastic/mock-sales/_search
[200] => {"took": 1, "timed_out": false, "_shards": {"total": 1, "successful": 1, "skipped": 0, "failed": 0},
GET /api/v1/_elastic/mock-sales/_field_caps?fields=quantity
[200] => {"indices": [], "fields": {}}
GET /api/v1/_elastic/mock-sales/_search?query=animal
[400] => {"message": "unknown field `query`, expected one of `_source`, `_source_excludes`, `_source_includes
GET /api/v1/_elastic/_search?q=animal
[501] => {"message": "_elastic/_search is not supported yet. Please try the index search endpoint (_elastic/{
GET /api/v1/indexes/mock-sales
[200] => {"version": "0.8", "index_uid": "mock-sales:01HZGCF383DMRQMFNEHPWFWB0S", "index_config": {"version":

[rdettai]

@tyrwzl we should add some instructions on how to set up Grafana on top of the API Gateway in the README. Would you be interested in opening a PR for that?

[tyrwzl]

@rdettai
Sure, I'll work on that.

[tyrwzl]

@rdettai
Hi, I created PR #5092.
Could you please take a look at it?