From 51e950e1436bb6c2974e323ed462f4b976b7f097 Mon Sep 17 00:00:00 2001 From: Liam Thompson <32779855+leemthompo@users.noreply.github.com> Date: Mon, 25 Nov 2024 23:18:51 +0100 Subject: [PATCH] Dedupe and clarify "Search your data" (#213) * Dedupe and clarify "Search your data" > Important: These docs now avoid duplicating content from core ES documentation to reduce maintenance overhead and prevent documentation drift. Instead, they focus on serverless-specific content and refer to core docs for implementation details. These docs were manually duplicated a looong time ago and are now incomplete and out of date. - Removed duplicated implementation guides - References core ES docs - Added tip for "bring your own vectors" tutorial - Removed duplicated implementation details - Points to core ES docs - Marked as duplicate/removed from navigation - Added Retrievers section - Simplified Query DSL explanation - References core ES docs - Rewritten for serverless context - Links to quickstart tutorials and notebooks - Points to core docs for concepts * Delete synonyms page (redirect is already in place) * fixes per review * Apply suggestions from code review Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> --------- Co-authored-by: shainaraskas <58563081+shainaraskas@users.noreply.github.com> --- .../index-serverless-elasticsearch.asciidoc | 1 - serverless/pages/knn-search.asciidoc | 1078 +---------------- .../pages/search-with-synonyms.asciidoc | 120 -- .../search-your-data-the-search-api.asciidoc | 23 +- serverless/pages/search-your-data.asciidoc | 35 +- 5 files changed, 44 insertions(+), 1213 deletions(-) delete mode 100644 serverless/pages/search-with-synonyms.asciidoc diff --git a/serverless/index-serverless-elasticsearch.asciidoc b/serverless/index-serverless-elasticsearch.asciidoc index 64d53024..a25e8c9c 100644 --- a/serverless/index-serverless-elasticsearch.asciidoc +++ b/serverless/index-serverless-elasticsearch.asciidoc @@ -37,7 +37,6 @@ include::./pages/ingest-your-data-ingest-data-through-integrations-beats.asciido include::./pages/search-your-data.asciidoc[leveloffset=+2] include::./pages/search-your-data-the-search-api.asciidoc[leveloffset=+3] -include::./pages/search-with-synonyms.asciidoc[leveloffset=+3] include::./pages/knn-search.asciidoc[leveloffset=+3] include::./pages/search-your-data-semantic-search.asciidoc[leveloffset=+3] diff --git a/serverless/pages/knn-search.asciidoc b/serverless/pages/knn-search.asciidoc index 0db335bc..b3d723fb 100644 --- a/serverless/pages/knn-search.asciidoc +++ b/serverless/pages/knn-search.asciidoc @@ -13,1081 +13,9 @@ Common use cases for kNN include: * Product recommendations and recommendation engines * Similarity search for images or videos -[discrete] -[[elasticsearch-knn-search-prerequisites]] -== Prerequisites - -* To run a kNN search, you must be able to convert your data into meaningful -vector values. You can -{ml-docs}/ml-nlp-text-emb-vector-search-example.html[create these vectors using -a natural language processing (NLP) model in {es}], or generate them outside -{es}. Vectors can be added to documents as {ref}/dense-vector.html[`dense_vector`] field -values. Queries are represented as vectors with the same dimension. -+ -Design your vectors so that the closer a document's vector is to a query vector, -based on a similarity metric, the better its match. -* To complete the steps in this guide, you must have the following -{ref}/security-privileges.html#privileges-list-indices[index privileges]: -+ -** `create_index` or `manage` to create an index with a `dense_vector` field -** `create`, `index`, or `write` to add data to the index you created -** `read` to search the index - -[discrete] -[[elasticsearch-knn-search-knn-methods]] -== kNN methods - -{es} supports two methods for kNN search: - -* Approximate kNN using the `knn` search -option -* Exact, brute-force kNN using a `script_score` query with a -vector function - -In most cases, you'll want to use approximate kNN. Approximate kNN offers lower -latency at the cost of slower indexing and imperfect accuracy. - -Exact, brute-force kNN guarantees accurate results but doesn't scale well with -large datasets. With this approach, a `script_score` query must scan each -matching document to compute the vector function, which can result in slow -search speeds. However, you can improve latency by using a {ref}/query-dsl.html[query] -to limit the number of matching documents passed to the function. If you -filter your data to a small subset of documents, you can get good search -performance using this approach. - -[discrete] -[[elasticsearch-knn-search-approximate-knn]] -== Approximate kNN - -To run an approximate kNN search, use the {ref}/knn-search.html#approximate-knn[`knn` option] -to search one or more `dense_vector` fields with indexing enabled. - -. Explicitly map one or more `dense_vector` fields. Approximate kNN search -requires the following mapping options: -+ -** A `similarity` value. This value determines the similarity metric used to -score documents based on similarity between the query and document vector. For a -list of available metrics, see the {ref}/dense-vector.html#dense-vector-similarity[`similarity`] -parameter documentation. The `similarity` setting defaults to `cosine`. -+ -[source,bash] ----- -curl -X PUT "${ES_URL}/image-index" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "mappings": { - "properties": { - "image-vector": { - "type": "dense_vector", - "dims": 3, - "similarity": "l2_norm" - }, - "title-vector": { - "type": "dense_vector", - "dims": 5, - "similarity": "l2_norm" - }, - "title": { - "type": "text" - }, - "file-type": { - "type": "keyword" - } - } - } -} -' ----- -. Index your data. -+ -[source,bash] ----- -curl -X POST "${ES_URL}/image-index/_bulk?refresh=true" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ "index": { "_id": "1" } } -{ "image-vector": [1, 5, -20], "title-vector": [12, 50, -10, 0, 1], "title": "moose family", "file-type": "jpg" } -{ "index": { "_id": "2" } } -{ "image-vector": [42, 8, -15], "title-vector": [25, 1, 4, -12, 2], "title": "alpine lake", "file-type": "png" } -{ "index": { "_id": "3" } } -{ "image-vector": [15, 11, 23], "title-vector": [1, 5, 25, 50, 20], "title": "full moon", "file-type": "jpg" } -... -' ----- -+ -// TEST[continued] -+ -// TEST[s/\.\.\.//] -. Run the search using the {ref}/knn-search.html#approximate-knn[`knn` option]. -+ -[source,bash] ----- -curl -X POST "${ES_URL}/image-index/_search" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "knn": { - "field": "image-vector", - "query_vector": [ - -5, - 9, - -12 - ], - "k": 10, - "num_candidates": 100 - }, - "fields": [ - "title", - "file-type" - ] -} -' ----- -+ -// TEST[continued] -+ -// TEST[s/"k": 10/"k": 3/] -+ -// TEST[s/"num_candidates": 100/"num_candidates": 3/] - -The {ref}/search-search.html#search-api-response-body-score[document `_score`] is determined by -the similarity between the query and document vector. See -{ref}/dense-vector.html#dense-vector-similarity[`similarity`] for more information on how kNN -search scores are computed. - -[discrete] -[[elasticsearch-knn-search-tune-approximate-knn-for-speed-or-accuracy]] -=== Tune approximate kNN for speed or accuracy - -To gather results, the kNN search API finds a `num_candidates` number of -approximate nearest neighbor candidates on each shard. The search computes the -similarity of these candidate vectors to the query vector, selecting the `k` -most similar results from each shard. The search then merges the results from -each shard to return the global top `k` nearest neighbors. - -You can increase `num_candidates` for more accurate results at the cost of -slower search speeds. A search with a high value for `num_candidates` -considers more candidates from each shard. This takes more time, but the -search has a higher probability of finding the true `k` top nearest neighbors. - -Similarly, you can decrease `num_candidates` for faster searches with -potentially less accurate results. - -[discrete] -[[approximate-knn-using-byte-vectors]] -=== Approximate kNN using byte vectors - -The approximate kNN search API supports `byte` value vectors in -addition to `float` value vectors. Use the {ref}/knn-search.html#approximate-knn[`knn` option] -to search a `dense_vector` field with {ref}/dense-vector.html#dense-vector-params[`element_type`] set to -`byte` and indexing enabled. - -. Explicitly map one or more `dense_vector` fields with -{ref}/dense-vector.html#dense-vector-params[`element_type`] set to `byte` and indexing enabled. -+ -[source,bash] ----- -curl -X PUT "${ES_URL}/byte-image-index" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "mappings": { - "properties": { - "byte-image-vector": { - "type": "dense_vector", - "element_type": "byte", - "dims": 2 - }, - "title": { - "type": "text" - } - } - } -} -' ----- -+ -// TEST[continued] -. Index your data ensuring all vector values -are integers within the range [-128, 127]. -+ -[source,bash] ----- -curl -X POST "${ES_URL}/byte-image-index/_bulk?refresh=true" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ "index": { "_id": "1" } } -{ "byte-image-vector": [5, -20], "title": "moose family" } -{ "index": { "_id": "2" } } -{ "byte-image-vector": [8, -15], "title": "alpine lake" } -{ "index": { "_id": "3" } } -{ "byte-image-vector": [11, 23], "title": "full moon" } -' ----- -+ -// TEST[continued] -. Run the search using the {ref}/knn-search.html#approximate-knn[`knn` option] -ensuring the `query_vector` values are integers within the -range [-128, 127]. -+ -[source,bash] ----- -curl -X POST "${ES_URL}/byte-image-index/_search" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "knn": { - "field": "byte-image-vector", - "query_vector": [ - -5, - 9 - ], - "k": 10, - "num_candidates": 100 - }, - "fields": [ - "title" - ] -} -' ----- -+ -// TEST[continued] -+ -// TEST[s/"k": 10/"k": 3/] -+ -// TEST[s/"num_candidates": 100/"num_candidates": 3/] - -[discrete] -[[elasticsearch-knn-search-filtered-knn-search]] -=== Filtered kNN search - -The kNN search API supports restricting the search using a filter. The search -will return the top `k` documents that also match the filter query. - -The following request performs an approximate kNN search filtered by the -`file-type` field: - -[source,bash] ----- -curl -X POST "${ES_URL}/image-index/_search" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "knn": { - "field": "image-vector", - "query_vector": [54, 10, -2], - "k": 5, - "num_candidates": 50, - "filter": { - "term": { - "file-type": "png" - } - } - }, - "fields": ["title"], - "_source": false -} -' ----- - -// TEST[continued] - -[NOTE] -==== -The filter is applied **during** the approximate kNN search to ensure -that `k` matching documents are returned. This contrasts with a -post-filtering approach, where the filter is applied **after** the approximate -kNN search completes. Post-filtering has the downside that it sometimes -returns fewer than k results, even when there are enough matching documents. -==== - -[discrete] -[[elasticsearch-knn-search-combine-approximate-knn-with-other-features]] -=== Combine approximate kNN with other features - -You can perform 'hybrid retrieval' by providing both the -{ref}/knn-search.html#approximate-knn[`knn` option] and a {ref}/search-search.html#request-body-search-query[`query`]: - -[source,bash] ----- -curl -X POST "${ES_URL}/image-index/_search" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "query": { - "match": { - "title": { - "query": "mountain lake", - "boost": 0.9 - } - } - }, - "knn": { - "field": "image-vector", - "query_vector": [54, 10, -2], - "k": 5, - "num_candidates": 50, - "boost": 0.1 - }, - "size": 10 -} -' ----- - -// TEST[continued] - -This search finds the global top `k = 5` vector matches, combines them with the matches from the `match` query, and -finally returns the 10 top-scoring results. The `knn` and `query` matches are combined through a disjunction, as if you -took a boolean 'or' between them. The top `k` vector results represent the global nearest neighbors across all index -shards. - -The score of each hit is the sum of the `knn` and `query` scores. You can specify a `boost` value to give a weight to -each score in the sum. In the example above, the scores will be calculated as - -[source,txt] ----- -score = 0.9 * match_score + 0.1 * knn_score ----- - -The `knn` option can also be used with {ref}/search-aggregations.html[aggregations]. -In general, {es} computes aggregations over all documents that match the search. -So for approximate kNN search, aggregations are calculated on the top `k` -nearest documents. If the search also includes a `query`, then aggregations are -calculated on the combined set of `knn` and `query` matches. - -[discrete] -[[elasticsearch-knn-search-perform-semantic-search]] -=== Perform semantic search - -kNN search enables you to perform semantic search by using a previously deployed -{ml-docs}/ml-nlp-search-compare.html#ml-nlp-text-embedding[text embedding model]. -Instead of literal matching on search terms, semantic search retrieves results -based on the intent and the contextual meaning of a search query. - -Under the hood, the text embedding NLP model generates a dense vector from the -input query string called `model_text` you provide. Then, it is searched -against an index containing dense vectors created with the same text embedding -{ml} model. The search results are semantically similar as learned by the model. - -[IMPORTANT] -==== -To perform semantic search: - -* you need an index that contains the dense vector representation of the input -data to search against, -* you must use the same text embedding model for search that you used to create -the dense vectors from the input data, -* the text embedding NLP model deployment must be started. -==== - -Reference the deployed text embedding model or the model deployment in the -`query_vector_builder` object and provide the search query as `model_text`: - -// NOTCONSOLE - -[source,js] ----- -(...) -{ - "knn": { - "field": "dense-vector-field", - "k": 10, - "num_candidates": 100, - "query_vector_builder": { - "text_embedding": { <1> - "model_id": "my-text-embedding-model", <2> - "model_text": "The opposite of blue" <3> - } - } - } -} -(...) ----- - -<1> The {nlp} task to perform. It must be `text_embedding`. - -<2> The ID of the text embedding model to use to generate the dense vectors from -the query string. Use the same model that generated the embeddings from the -input text in the index you search against. You can use the value of the -`deployment_id` instead in the `model_id` argument. - -<3> The query string from which the model generates the dense vector -representation. - -For more information on how to deploy a trained model and use it to create text -embeddings, refer to this -{ml-docs}/ml-nlp-text-emb-vector-search-example.html[end-to-end example]. - -[discrete] -[[elasticsearch-knn-search-search-multiple-knn-fields]] -=== Search multiple kNN fields - -In addition to 'hybrid retrieval', you can search more than one kNN vector field at a time: - -[source,bash] ----- -curl -X POST "${ES_URL}/image-index/_search" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "query": { - "match": { - "title": { - "query": "mountain lake", - "boost": 0.9 - } - } - }, - "knn": [ { - "field": "image-vector", - "query_vector": [54, 10, -2], - "k": 5, - "num_candidates": 50, - "boost": 0.1 - }, - { - "field": "title-vector", - "query_vector": [1, 20, -52, 23, 10], - "k": 10, - "num_candidates": 10, - "boost": 0.5 - }], - "size": 10 -} -' ----- - -// TEST[continued] - -This search finds the global top `k = 5` vector matches for `image-vector` and the global `k = 10` for the `title-vector`. -These top values are then combined with the matches from the `match` query and the top-10 documents are returned. -The multiple `knn` entries and the `query` matches are combined through a disjunction, -as if you took a boolean 'or' between them. The top `k` vector results represent the global nearest neighbors across -all index shards. - -The scoring for a doc with the above configured boosts would be: - -[source,txt] ----- -score = 0.9 * match_score + 0.1 * knn_score_image-vector + 0.5 * knn_score_title-vector ----- - -[discrete] -[[elasticsearch-knn-search-search-knn-with-expected-similarity]] -=== Search kNN with expected similarity - -While kNN is a powerful tool, it always tries to return `k` nearest neighbors. Consequently, when using `knn` with -a `filter`, you could filter out all relevant documents and only have irrelevant ones left to search. In that situation, -`knn` will still do its best to return `k` nearest neighbors, even though those neighbors could be far away in the -vector space. - -To alleviate this worry, there is a `similarity` parameter available in the `knn` clause. This value is the required -minimum similarity for a vector to be considered a match. The `knn` search flow with this parameter is as follows: - -* Apply any user provided `filter` queries -* Explore the vector space to get `k` vectors -* Do not return any vectors that are further away than the configured `similarity` - -Here is an example. In this example we search for the given `query_vector` for `k` nearest neighbors. However, with -`filter` applied and requiring that the found vectors have at least the provided `similarity` between them. - -[source,bash] ----- -curl -X POST "${ES_URL}/image-index/_search" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "knn": { - "field": "image-vector", - "query_vector": [1, 5, -20], - "k": 5, - "num_candidates": 50, - "similarity": 36, - "filter": { - "term": { - "file-type": "png" - } - } - }, - "fields": ["title"], - "_source": false -} -' ----- - -// TEST[continued] - -In our data set, the only document with the file type of `png` has a vector of `[42, 8, -15]`. The `l2_norm` distance -between `[42, 8, -15]` and `[1, 5, -20]` is `41.412`, which is greater than the configured similarity of `36`. Meaning, -this search will return no hits. - -[discrete] -[[nested-knn-search]] -=== Nested kNN Search - -It is common for text to exceed a particular model's token limit and requires chunking before building the embeddings -for individual chunks. When using {ref}/nested.html[`nested`] with {ref}/dense-vector.html[`dense_vector`], you can achieve nearest -passage retrieval without copying top-level document metadata. - -Here is a simple passage vectors index that stores vectors and some top-level metadata for filtering. - -[source,bash] ----- -curl -X PUT "${ES_URL}/passage_vectors" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "mappings": { - "properties": { - "full_text": { - "type": "text" - }, - "creation_time": { - "type": "date" - }, - "paragraph": { - "type": "nested", - "properties": { - "vector": { - "type": "dense_vector", - "dims": 2 - }, - "text": { - "type": "text", - "index": false - } - } - } - } - } -} -' ----- - -// TEST[continued] - -With the above mapping, we can index multiple passage vectors along with storing the individual passage text. - -[source,bash] ----- -curl -X POST "${ES_URL}/passage_vectors/_bulk?refresh=true" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ "index": { "_id": "1" } } -{ "full_text": "first paragraph another paragraph", "creation_time": "2019-05-04", "paragraph": [ { "vector": [ 0.45, 45 ], "text": "first paragraph", "paragraph_id": "1" }, { "vector": [ 0.8, 0.6 ], "text": "another paragraph", "paragraph_id": "2" } ] } -{ "index": { "_id": "2" } } -{ "full_text": "number one paragraph number two paragraph", "creation_time": "2020-05-04", "paragraph": [ { "vector": [ 1.2, 4.5 ], "text": "number one paragraph", "paragraph_id": "1" }, { "vector": [ -1, 42 ], "text": "number two paragraph", "paragraph_id": "2" } ] } -' ----- - -// TEST[continued] - -// TEST[s/\.\.\.//] - -The query will seem very similar to a typical kNN search: - -[source,bash] ----- -curl -X POST "${ES_URL}/passage_vectors/_search" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "fields": ["full_text", "creation_time"], - "_source": false, - "knn": { - "query_vector": [ - 0.45, - 45 - ], - "field": "paragraph.vector", - "k": 2, - "num_candidates": 2 - } -} -' ----- - -// TEST[continued] - -Note below that even though we have 4 total vectors, we still return two documents. kNN search over nested dense_vectors -will always diversify the top results over the top-level document. Meaning, `"k"` top-level documents will be returned, -scored by their nearest passage vector (e.g. `"paragraph.vector"`). - -[source,console-result] ----- -{ - "took": 4, - "timed_out": false, - "_shards": { - "total": 1, - "successful": 1, - "skipped": 0, - "failed": 0 - }, - "hits": { - "total": { - "value": 2, - "relation": "eq" - }, - "max_score": 1.0, - "hits": [ - { - "_index": "passage_vectors", - "_id": "1", - "_score": 1.0, - "fields": { - "creation_time": [ - "2019-05-04T00:00:00.000Z" - ], - "full_text": [ - "first paragraph another paragraph" - ] - } - }, - { - "_index": "passage_vectors", - "_id": "2", - "_score": 0.9997144, - "fields": { - "creation_time": [ - "2020-05-04T00:00:00.000Z" - ], - "full_text": [ - "number one paragraph number two paragraph" - ] - } - } - ] - } -} ----- - -// TESTRESPONSE[s/"took": 4/"took" : "$body.took"/] - -What if you wanted to filter by some top-level document metadata? You can do this by adding `filter` to your -`knn` clause. - -[NOTE] -==== -`filter` will always be over the top-level document metadata. This means you cannot filter based on `nested` -field metadata. -==== - -[source,bash] ----- -curl -X POST "${ES_URL}/passage_vectors/_search" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "fields": [ - "creation_time", - "full_text" - ], - "_source": false, - "knn": { - "query_vector": [ - 0.45, - 45 - ], - "field": "paragraph.vector", - "k": 2, - "num_candidates": 2, - "filter": { - "bool": { - "filter": [ - { - "range": { - "creation_time": { - "gte": "2019-05-01", - "lte": "2019-05-05" - } - } - } - ] - } - } - } -} -' ----- - -// TEST[continued] - -Now we have filtered based on the top level `"creation_time"` and only one document falls within that range. - -[source,console-result] ----- -{ - "took": 4, - "timed_out": false, - "_shards": { - "total": 1, - "successful": 1, - "skipped": 0, - "failed": 0 - }, - "hits": { - "total": { - "value": 1, - "relation": "eq" - }, - "max_score": 1.0, - "hits": [ - { - "_index": "passage_vectors", - "_id": "1", - "_score": 1.0, - "fields": { - "creation_time": [ - "2019-05-04T00:00:00.000Z" - ], - "full_text": [ - "first paragraph another paragraph" - ] - } - } - ] - } -} ----- - -// TESTRESPONSE[s/"took": 4/"took" : "$body.took"/] - -Additionally, if you wanted to extract the nearest passage for a matched document, you can supply {ref}/inner-hits.html[inner_hits] -to the `knn` clause. - -[NOTE] -==== -`inner_hits` for kNN will only ever return a single hit, the nearest passage vector. -Setting `"size"` to any value greater than `1` will have no effect on the results. -==== - -[source,bash] ----- -curl -X POST "${ES_URL}/passage_vectors/_search" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "fields": [ - "creation_time", - "full_text" - ], - "_source": false, - "knn": { - "query_vector": [ - 0.45, - 45 - ], - "field": "paragraph.vector", - "k": 2, - "num_candidates": 2, - "inner_hits": { - "_source": false, - "fields": [ - "paragraph.text" - ] - } - } -} -' ----- - -// TEST[continued] - -Now the result will contain the nearest found paragraph when searching. - -[source,console-result] ----- -{ - "took": 4, - "timed_out": false, - "_shards": { - "total": 1, - "successful": 1, - "skipped": 0, - "failed": 0 - }, - "hits": { - "total": { - "value": 2, - "relation": "eq" - }, - "max_score": 1.0, - "hits": [ - { - "_index": "passage_vectors", - "_id": "1", - "_score": 1.0, - "fields": { - "creation_time": [ - "2019-05-04T00:00:00.000Z" - ], - "full_text": [ - "first paragraph another paragraph" - ] - }, - "inner_hits": { - "paragraph": { - "hits": { - "total": { - "value": 1, - "relation": "eq" - }, - "max_score": 1.0, - "hits": [ - { - "_index": "passage_vectors", - "_id": "1", - "_nested": { - "field": "paragraph", - "offset": 0 - }, - "_score": 1.0, - "fields": { - "paragraph": [ - { - "text": [ - "first paragraph" - ] - } - ] - } - } - ] - } - } - } - }, - { - "_index": "passage_vectors", - "_id": "2", - "_score": 0.9997144, - "fields": { - "creation_time": [ - "2020-05-04T00:00:00.000Z" - ], - "full_text": [ - "number one paragraph number two paragraph" - ] - }, - "inner_hits": { - "paragraph": { - "hits": { - "total": { - "value": 1, - "relation": "eq" - }, - "max_score": 0.9997144, - "hits": [ - { - "_index": "passage_vectors", - "_id": "2", - "_nested": { - "field": "paragraph", - "offset": 1 - }, - "_score": 0.9997144, - "fields": { - "paragraph": [ - { - "text": [ - "number two paragraph" - ] - } - ] - } - } - ] - } - } - } - } - ] - } -} ----- - -// TESTRESPONSE[s/"took": 4/"took" : "$body.took"/] - -[discrete] -[[knn-indexing-considerations]] -=== Indexing considerations - -For approximate kNN search, {es} stores the dense vector values of each -segment as an https://arxiv.org/abs/1603.09320[HNSW graph]. Indexing vectors for -approximate kNN search can take substantial time because of how expensive it is -to build these graphs. You may need to increase the client request timeout for -index and bulk requests. The {ref}/tune-knn-search.html[approximate kNN tuning guide] -contains important guidance around indexing performance, and how the index -configuration can affect search performance. - -In addition to its search-time tuning parameters, the HNSW algorithm has -index-time parameters that trade off between the cost of building the graph, -search speed, and accuracy. When setting up the `dense_vector` mapping, you -can use the {ref}/dense-vector.html#dense-vector-index-options[`index_options`] argument to adjust -these parameters: - -[source,bash] ----- -curl -X PUT "${ES_URL}/image-index" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "mappings": { - "properties": { - "image-vector": { - "type": "dense_vector", - "dims": 3, - "index": true, - "similarity": "l2_norm", - "index_options": { - "type": "hnsw", - "m": 32, - "ef_construction": 100 - } - } - } - } -} -' ----- - -[discrete] -[[elasticsearch-knn-search-limitations-for-approximate-knn-search]] -=== Limitations for approximate kNN search - -{es} uses the https://arxiv.org/abs/1603.09320[HNSW algorithm] to support -efficient kNN search. Like most kNN algorithms, HNSW is an approximate method -that sacrifices result accuracy for improved search speed. This means the -results returned are not always the true _k_ closest neighbors. - -[NOTE] -==== -Approximate kNN search always uses the -{ref}/search-search.html#dfs-query-then-fetch[`dfs_query_then_fetch`] search type in order to gather -the global top `k` matches across shards. You cannot set the -`search_type` explicitly when running kNN search. -==== - -[discrete] -[[exact-knn]] -== Exact kNN - -To run an exact kNN search, use a `script_score` query with a vector function. - -. Explicitly map one or more `dense_vector` fields. If you don't intend to use -the field for approximate kNN, set the `index` mapping option to `false`. -This can significantly improve indexing speed. -+ -[source,bash] ----- -curl -X PUT "${ES_URL}/product-index" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "mappings": { - "properties": { - "product-vector": { - "type": "dense_vector", - "dims": 5, - "index": false - }, - "price": { - "type": "long" - } - } - } -} -' ----- -. Index your data. -+ -[source,bash] ----- -curl -X POST "${ES_URL}/product-index/_bulk?refresh=true" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ "index": { "_id": "1" } } -{ "product-vector": [230.0, 300.33, -34.8988, 15.555, -200.0], "price": 1599 } -{ "index": { "_id": "2" } } -{ "product-vector": [-0.5, 100.0, -13.0, 14.8, -156.0], "price": 799 } -{ "index": { "_id": "3" } } -{ "product-vector": [0.5, 111.3, -13.0, 14.8, -156.0], "price": 1099 } -... -' ----- -+ -// TEST[continued] -+ -// TEST[s/\.\.\.//] -. Use the search API to run a `script_score` query containing -a {ref}/query-dsl-script-score-query.html#vector-functions[vector function]. +Learn more in the {ref}/knn-search.html[{es} core documentation]. [TIP] ==== -To limit the number of matched documents passed to the vector function, we -recommend you specify a filter query in the `script_score.query` parameter. If -needed, you can use a {ref}/query-dsl-match-all-query.html[`match_all` query] in this -parameter to match all documents. However, matching all documents can -significantly increase search latency. - -[source,bash] ----- -curl -X POST "${ES_URL}/product-index/_search" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d ' -{ - "query": { - "script_score": { - "query": { - "bool": { - "filter": { - "range": { - "price": { - "gte": 1000 - } - } - } - } - }, - "script": { - "source": "cosineSimilarity(params.queryVector, 'product-vector') + 1.0", - "params": { - "queryVector": [ - -0.5, - 90, - -10, - 14.8, - -156 - ] - } - } - } - } -} -' ----- - -// TEST[continued] -==== +Check out our {ref}/bring-your-own-vectors.html[hands-on tutorial] to learn how to ingest dense vector embeddings into Elasticsearch. +==== \ No newline at end of file diff --git a/serverless/pages/search-with-synonyms.asciidoc b/serverless/pages/search-with-synonyms.asciidoc deleted file mode 100644 index e1075134..00000000 --- a/serverless/pages/search-with-synonyms.asciidoc +++ /dev/null @@ -1,120 +0,0 @@ -[[elasticsearch-reference-search-with-synonyms]] -= Full-text search with synonyms - -// :description: Use synonyms to search for words or phrases that have the same or similar meaning. -// :keywords: serverless, elasticsearch, search, synonyms - -Synonyms are words or phrases that have the same or similar meaning. -They are an important aspect of search, as they can improve the search experience and increase the scope of search results. - -Synonyms allow you to: - -* **Improve search relevance** by finding relevant documents that use different terms to express the same concept. -* Make **domain-specific vocabulary** more user-friendly, allowing users to use search terms they are more familiar with. -* **Define common misspellings and typos** to transparently handle common mistakes. - -Synonyms are grouped together using **synonyms sets**. -You can have as many synonyms sets as you need. - -In order to use synonyms sets in {es}, you need to: - -* Store your synonyms set -* Configure synonyms token filters and analyzers - -[discrete] -[[elasticsearch-reference-search-with-synonyms-store-your-synonyms-set]] -== Store your synonyms set - -Your synonyms sets need to be stored in {es} so your analyzers can refer to them. -There are two ways to store your synonyms sets: - -[discrete] -[[elasticsearch-reference-search-with-synonyms-synonyms-api]] -=== Synonyms API - -You can use the {ref}/synonyms-apis.html[synonyms APIs] to manage synonyms sets. -This is the most flexible approach, as it allows to dynamically define and modify synonyms sets. - -Changes in your synonyms sets will automatically reload the associated analyzers. - -[discrete] -[[elasticsearch-reference-search-with-synonyms-inline]] -=== Inline - -You can test your synonyms by adding them directly inline in your token filter definition. - -[WARNING] -==== -Inline synonyms are not recommended for production usage. -A large number of inline synonyms increases cluster size unnecessarily and can lead to performance issues. -==== - -[discrete] -[[synonyms-synonym-token-filters]] -=== Configure synonyms token filters and analyzers - -Once your synonyms sets are created, you can start configuring your token filters and analyzers to use them. - -{es} uses synonyms as part of the {ref}/analysis-overview.html[analysis process]. -You can use two types of {ref}/analysis-tokenfilters.html[token filter]: - -* {ref}/analysis-synonym-graph-tokenfilter.html[Synonym graph token filter]: It is recommended to use it, as it can correctly handle multi-word synonyms ("hurriedly", "in a hurry"). -* {ref}/analysis-synonym-tokenfilter.html[Synonym token filter]: Not recommended if you need to use multi-word synonyms. - -Check each synonym token filter documentation for configuration details and instructions on adding it to an analyzer. - -[discrete] -[[elasticsearch-reference-search-with-synonyms-test-your-analyzer]] -=== Test your analyzer - -You can test an analyzer configuration without modifying your index settings. -Use the {ref}/indices-analyze.html[analyze API] to test your analyzer chain: - -[source,bash] ----- -curl "${ES_URL}/my-index/_analyze?pretty" \ --H "Authorization: ApiKey ${API_KEY}" \ --H "Content-Type: application/json" \ --d' -{ - "tokenizer": "standard", - "filter" : [ - "lowercase", - { - "type": "synonym_graph", - "synonyms": ["pc => personal computer", "computer, pc, laptop"] - } - ], - "text" : "Check how PC synonyms work" -} ----- - -[discrete] -[[elasticsearch-reference-search-with-synonyms-apply-synonyms-at-index-or-search-time]] -=== Apply synonyms at index or search time - -Analyzers can be applied at {ref}/analysis-index-search-time.html[index time or search time]. - -You need to decide when to apply your synonyms: - -* Index time: Synonyms are applied when the documents are indexed into {es}. This is a less flexible alternative, as changes to your synonyms require {ref}/docs-reindex.html[reindexing]. -* Search time: Synonyms are applied when a search is executed. This is a more flexible approach, which doesn't require reindexing. If token filters are configured with `"updateable": true`, search analyzers can be {ref}/indices-reload-analyzers.html[reloaded] when you make changes to your synonyms. - -Synonyms sets created using the synonyms API can only be used at search time. - -You can specify the analyzer that contains your synonyms set as a {ref}/specify-analyzer.html#specify-search-analyzer[search time analyzer] or as an {ref}/specify-analyzer.html#specify-index-time-analyzer[index time analyzer]. - -The following example adds `my_analyzer` as a search analyzer to the `title` field in an index mapping: - -[source,JSON] ----- - "mappings": { - "properties": { - "title": { - "type": "text", - "search_analyzer": "my_analyzer", - "updateable": true - } - } - } ----- diff --git a/serverless/pages/search-your-data-the-search-api.asciidoc b/serverless/pages/search-your-data-the-search-api.asciidoc index dcf65a95..f0a0f29c 100644 --- a/serverless/pages/search-your-data-the-search-api.asciidoc +++ b/serverless/pages/search-your-data-the-search-api.asciidoc @@ -14,7 +14,24 @@ a specific number of results. You can use the https://www.elastic.co/docs/api/doc/elasticsearch-serverless/group/endpoint-search[search API] to search and aggregate data stored in {es} data streams or indices. -The API's `query` request body parameter accepts queries written in -{ref}/query-dsl.html[Query DSL]. -For more information, refer to {ref}/search-your-data.html[the search API overview] in the classic {es} docs. +For more information, refer to {ref}/search-your-data.html[the search API overview] in the core {es} docs. + +[discrete] +[[elasticsearch-search-your-data-the-query-dsl]] +== Query DSL + +{ref}/query-dsl.html[Query DSL] is full-featured JSON-style query language that enables complex searching, filtering, and aggregations. + +The `_search` API's `query` request body parameter accepts queries written in +Query DSL. + +[discrete] +[[elasticsearch-search-your-data-the-retrievers]] +== Retrievers + +Retrievers are an alternative to Query DSL that allow you to configure complex retrieval pipelines using a simplified syntax. +Retrievers simplify the user experience by allowing entire retrieval pipelines to be configured in a single `_search` API call. + +Learn more in the {ref}/retrievers-overview.html[Retrievers overview] in the core {es} docs. + diff --git a/serverless/pages/search-your-data.asciidoc b/serverless/pages/search-your-data.asciidoc index 196e5b1c..194e88c4 100644 --- a/serverless/pages/search-your-data.asciidoc +++ b/serverless/pages/search-your-data.asciidoc @@ -1,26 +1,33 @@ [[elasticsearch-search-your-data]] = Search your data -// :description: Use the search API to run queries on your data. +// :description: Learn options for searching your data in Elasticsearch Serverless. // :keywords: serverless, elasticsearch, search -A search query, or query, is a request for information about data in {es} data streams or indices. +Searching your data in {es-serverless} works the same way as in other Elasticsearch deployments. +If you haven't used {es} before, you can learn the basics in the {ref}/elasticsearch-intro.html[core {es} documentation]. -You can think of a query as a question, written in a way {es} understands. Depending on your data, you can use a query to get answers to questions like: +You can use the https://www.elastic.co/docs/api/doc/elasticsearch-serverless[{es-serverless} REST APIs] to search your data using any HTTP client, including the <>, or directly in <>. -* What processes on my server take longer than 500 milliseconds to respond? -* What users on my network ran regsvr32.exe within the last week? -* What pages on my website contain a specific word or phrase? +You can also run searches using {kibana-ref}/discover.html[Discover] in your project's UI. -You run search queries using the <>. The API supports several query types and search methods: +[TIP] +==== +Try our hands-on {ref}/quickstart.html#quickstart-list[quick start tutorials] in the core {es} documentation to get started, or check out our https://github.com/elastic/elasticsearch-labs/tree/main/notebooks#readme[Python notebooks]. +==== -**Search for exact values.** -Use {ref}/term-level-queries.html[term-level queries] to filter numbers, dates, IPs, or strings based on exact values or ranges. +[discrete] +[[elasticsearch-search-your-data-query-languages-overview]] +== Query languages -**Full-text search.** -Use {ref}/full-text-queries.html[full-text queries] to query {ref}/analysis.html#analysis[unstructured text] and find documents that best match query terms. Use <> to search for words or phrases that have the same or similar meaning. +Learn about the various query languages you can use to search your data in the {ref}/search-analyze.html[core {es} documentation]. -**Vector search.** -Store dense vectors in {es} and use <> to find similar vectors. +[discrete] +[[elasticsearch-search-your-data-learn-more]] +== Learn more -You can also use Elastic's natural language processing (NLP) model to encode text as sparse or dense vectors. Then use <> to find data based on the intent and contextual meaning rather than matching keywords. +The subpages in this section provide some high-level information about selected important search-related topics: + +* <> +* <> +* <> \ No newline at end of file