From e1002d373267ab4a6e50ffe931f939018acbeebf Mon Sep 17 00:00:00 2001 From: Liam Thompson <32779855+leemthompo@users.noreply.github.com> Date: Thu, 17 Oct 2024 19:01:46 +0200 Subject: [PATCH 1/4] Update links to search connectors docs (#4389) (cherry picked from commit 518039675b3be9f612b236653679958c6d690644) # Conflicts: # docs/en/observability/observability-ai-assistant.asciidoc --- .../observability-ai-assistant.asciidoc | 45 +++++++++++++++++++ 1 file changed, 45 insertions(+) diff --git a/docs/en/observability/observability-ai-assistant.asciidoc b/docs/en/observability/observability-ai-assistant.asciidoc index 441e00e30d..1f3f9262ba 100644 --- a/docs/en/observability/observability-ai-assistant.asciidoc +++ b/docs/en/observability/observability-ai-assistant.asciidoc @@ -39,6 +39,10 @@ The AI assistant requires the following: ** OpenAI `gpt-4`+. ** Azure OpenAI Service `gpt-4`(0613) or `gpt-4-32k`(0613) with API version `2023-07-01-preview` or more recent. ** AWS Bedrock, specifically the Anthropic Claude models. +<<<<<<< HEAD +======= +* An {enterprise-search-ref}/server.html[Enterprise Search] server if {ref}/es-connectors.html[search connectors] are used to populate external data into the knowledge base. +>>>>>>> 51803967 (Update links to search connectors docs (#4389)) * The knowledge base requires a 4 GB {ml} node. [IMPORTANT] @@ -144,6 +148,7 @@ To add external data to the knowledge base in {kib}: ** `user.name` — (can be `null`) if defined, restricts the internal document's availability to a specific user. ** You can add a query filter to index specific documents. +<<<<<<< HEAD [source,console] ---- POST _reindex @@ -168,6 +173,46 @@ POST _reindex } } ---- +======= +{enterprise-search-ref}/connectors.html[Connectors] allow you to index content from external sources thereby making it available for the AI Assistant. This can greatly improve the relevance of the AI Assistant’s responses. Data can be integrated from sources such as GitHub, Confluence, Google Drive, Jira, AWS S3, Microsoft Teams, Slack, and more. + +These connectors are managed under *Search* -> *Content* -> *Connectors* in {kib}, they are outside of the {observability} Solution, and they require an {enterprise-search-ref}/server.html[Enterprise Search] server connected to the Elastic Stack. + +By default, the AI Assistant queries all search connector indices. To override this behavior and customize which indices are queried, adjust the *Search connector index pattern* setting on the <> page. This allows precise control over which data sources are included in AI Assistant knowledge base. + +To create a connector and make its content available to the AI Assistant knowledge base, follow these steps: + +. In {kib} UI, go to *Search* -> *Content* -> *Connectors* and follow the instructions to create a new connector. ++ +[NOTE] +==== +If your {kib} Space doesn't include the `Search` solution you will have to create the connector from a different space or change your space *Solution view* setting to `Classic`. +==== ++ +For example, if you create a {ref}/es-connectors-github.html[GitHub connector] you have to set a `name`, attach it to a new or existing `index`, add your `personal access token` and include the `list of repositories` to synchronize. ++ +Learn more about configuring and {ref}/es-connectors-usage.html[using connectors] in the Elasticsearch documentation. ++ +. Create a pipeline and process the data with ELSER. ++ +To create the embeddings needed by the AI Assistant (weights and tokens into a sparse vector field), you have to create an *ML Inference Pipeline*: ++ +.. Open the previously created connector and select the *Pipelines* tab. +.. Select *Copy and customize* button at the `Unlock your custom pipelines` box. +.. Select *Add Inference Pipeline* button at the `Machine Learning Inference Pipelines` box. +.. Select *ELSER (Elastic Learned Sparse EncodeR)* ML model to add the necessary embeddings to the data. +.. Select the fields that need to be evaluated as part of the inference pipeline. +.. Test and save the inference pipeline and the overall pipeline. +. Sync the data. ++ +Once the pipeline is set up, perform a *Full Content Sync* of the connector. The inference pipeline will process the data as follows: ++ +* As data comes in, ELSER is applied to the data, and embeddings (weights and tokens into a sparse vector field) are added to capture semantic meaning and context of the data. +* When you look at the documents that are ingested, you can see how the weights and token are added to the `predicted_value` field in the documents. +. Check if AI Assistant can use the index (optional). ++ +Ask something to the AI Assistant related with the indexed data. +>>>>>>> 51803967 (Update links to search connectors docs (#4389)) [discrete] [[obs-ai-interact]] From eaf5fb94c03922d364b8a84b9cc322120f016a83 Mon Sep 17 00:00:00 2001 From: Brandon Morelli Date: Fri, 18 Oct 2024 12:29:32 -0700 Subject: [PATCH 2/4] Update observability-ai-assistant.asciidoc --- docs/en/observability/observability-ai-assistant.asciidoc | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/docs/en/observability/observability-ai-assistant.asciidoc b/docs/en/observability/observability-ai-assistant.asciidoc index 1f3f9262ba..6721cc0cb4 100644 --- a/docs/en/observability/observability-ai-assistant.asciidoc +++ b/docs/en/observability/observability-ai-assistant.asciidoc @@ -39,10 +39,7 @@ The AI assistant requires the following: ** OpenAI `gpt-4`+. ** Azure OpenAI Service `gpt-4`(0613) or `gpt-4-32k`(0613) with API version `2023-07-01-preview` or more recent. ** AWS Bedrock, specifically the Anthropic Claude models. -<<<<<<< HEAD -======= * An {enterprise-search-ref}/server.html[Enterprise Search] server if {ref}/es-connectors.html[search connectors] are used to populate external data into the knowledge base. ->>>>>>> 51803967 (Update links to search connectors docs (#4389)) * The knowledge base requires a 4 GB {ml} node. [IMPORTANT] @@ -148,7 +145,6 @@ To add external data to the knowledge base in {kib}: ** `user.name` — (can be `null`) if defined, restricts the internal document's availability to a specific user. ** You can add a query filter to index specific documents. -<<<<<<< HEAD [source,console] ---- POST _reindex @@ -173,7 +169,7 @@ POST _reindex } } ---- -======= + {enterprise-search-ref}/connectors.html[Connectors] allow you to index content from external sources thereby making it available for the AI Assistant. This can greatly improve the relevance of the AI Assistant’s responses. Data can be integrated from sources such as GitHub, Confluence, Google Drive, Jira, AWS S3, Microsoft Teams, Slack, and more. These connectors are managed under *Search* -> *Content* -> *Connectors* in {kib}, they are outside of the {observability} Solution, and they require an {enterprise-search-ref}/server.html[Enterprise Search] server connected to the Elastic Stack. @@ -212,7 +208,6 @@ Once the pipeline is set up, perform a *Full Content Sync* of the connector. The . Check if AI Assistant can use the index (optional). + Ask something to the AI Assistant related with the indexed data. ->>>>>>> 51803967 (Update links to search connectors docs (#4389)) [discrete] [[obs-ai-interact]] From 3cd7c0c833cc39a193cd296d5ac770ea4041ff84 Mon Sep 17 00:00:00 2001 From: Brandon Morelli Date: Fri, 18 Oct 2024 12:31:47 -0700 Subject: [PATCH 3/4] Update docs/en/observability/observability-ai-assistant.asciidoc --- .../observability-ai-assistant.asciidoc | 39 ------------------- 1 file changed, 39 deletions(-) diff --git a/docs/en/observability/observability-ai-assistant.asciidoc b/docs/en/observability/observability-ai-assistant.asciidoc index a0f70e5ae8..acc2a8a5b2 100644 --- a/docs/en/observability/observability-ai-assistant.asciidoc +++ b/docs/en/observability/observability-ai-assistant.asciidoc @@ -183,45 +183,6 @@ Once the pipeline is set up, perform a *Full Content Sync* of the connector. The + Ask something to the AI Assistant related with the indexed data. -{enterprise-search-ref}/connectors.html[Connectors] allow you to index content from external sources thereby making it available for the AI Assistant. This can greatly improve the relevance of the AI Assistant’s responses. Data can be integrated from sources such as GitHub, Confluence, Google Drive, Jira, AWS S3, Microsoft Teams, Slack, and more. - -These connectors are managed under *Search* -> *Content* -> *Connectors* in {kib}, they are outside of the {observability} Solution, and they require an {enterprise-search-ref}/server.html[Enterprise Search] server connected to the Elastic Stack. - -By default, the AI Assistant queries all search connector indices. To override this behavior and customize which indices are queried, adjust the *Search connector index pattern* setting on the <> page. This allows precise control over which data sources are included in AI Assistant knowledge base. - -To create a connector and make its content available to the AI Assistant knowledge base, follow these steps: - -. In {kib} UI, go to *Search* -> *Content* -> *Connectors* and follow the instructions to create a new connector. -+ -[NOTE] -==== -If your {kib} Space doesn't include the `Search` solution you will have to create the connector from a different space or change your space *Solution view* setting to `Classic`. -==== -+ -For example, if you create a {ref}/es-connectors-github.html[GitHub connector] you have to set a `name`, attach it to a new or existing `index`, add your `personal access token` and include the `list of repositories` to synchronize. -+ -Learn more about configuring and {ref}/es-connectors-usage.html[using connectors] in the Elasticsearch documentation. -+ -. Create a pipeline and process the data with ELSER. -+ -To create the embeddings needed by the AI Assistant (weights and tokens into a sparse vector field), you have to create an *ML Inference Pipeline*: -+ -.. Open the previously created connector and select the *Pipelines* tab. -.. Select *Copy and customize* button at the `Unlock your custom pipelines` box. -.. Select *Add Inference Pipeline* button at the `Machine Learning Inference Pipelines` box. -.. Select *ELSER (Elastic Learned Sparse EncodeR)* ML model to add the necessary embeddings to the data. -.. Select the fields that need to be evaluated as part of the inference pipeline. -.. Test and save the inference pipeline and the overall pipeline. -. Sync the data. -+ -Once the pipeline is set up, perform a *Full Content Sync* of the connector. The inference pipeline will process the data as follows: -+ -* As data comes in, ELSER is applied to the data, and embeddings (weights and tokens into a sparse vector field) are added to capture semantic meaning and context of the data. -* When you look at the documents that are ingested, you can see how the weights and token are added to the `predicted_value` field in the documents. -. Check if AI Assistant can use the index (optional). -+ -Ask something to the AI Assistant related with the indexed data. - [discrete] [[obs-ai-interact]] == Interact with the AI Assistant From c59b544fe46a0edddc7fa04d6fb5065d594fa748 Mon Sep 17 00:00:00 2001 From: Brandon Morelli Date: Fri, 18 Oct 2024 13:48:01 -0700 Subject: [PATCH 4/4] retry ci