[Security solution] AWS Bedrock connector #166662

stephmilovic · 2023-09-18T18:49:41Z

Summary

New Generative AI connector to support AWS Bedrock. This change will need name change follow up as described here: #166960

Implemented with the sub actions framework. The actions are runApi and invokeAI. runApi is the action attached to the test sub action. runApi is responsible for making a POST request to the Bedrock API endpoint and returning the response data. This is an example payload of what Bedrock is expecting:

{
  "prompt": "\n\Human: Hello world! \n\nAssistant:",
  "max_tokens_to_sample": 300,
  "stop_sequences": ["\n\nAssistant:"],
}

Create connector

Test connector

Testing for ResponseOps team

My apologies for doing this all in one PR. Hopefully clear testing expectations will win me points back?
Bedrock credentials
ResponseOps should test that a connector can be created and tested from the Stack Management > Connectors page

Create Bedrock connectors with Bedrock credentials
Test the connector by sending example payload
Confirm results in Network tab. Should look like:

Testing for Security Solution team

Bedrock credentials
Security Solution should test that a connectors can be created 3 places within the AI assistant, that multiple connectors can be used in a single conversation, and that the connectors work with both Langchain and non-langchain:

Start fresh with no connectors and a cleared local storage (no conversation history)
Through the welcome conversation, create a Bedrock connector using the button in the conversation:

Send a message to confirm the connection
Create another Bedrock connector from the top left connector selector

Create an OpenAI/GenA connector from the settings

Ensure that the connectors are listed with the proper description:

Switch between connectors and have a chat with each, ensure that we can switch between Bedrock and OpenAI without issues
If you've had the setting assistantLangChain={false}, switch it to true or vice versa. You do not need to test the connector creation with langchain and non-langchain, but test sending messages with both langchain and non-langchain to both connector types

…-fix'

stephmilovic · 2023-09-20T22:25:23Z

package.json

@@ -838,6 +838,7 @@
    "antlr4ts": "^0.5.0-alpha.3",
    "archiver": "^5.3.1",
    "async": "^3.2.3",
+    "aws4": "^1.12.0",


need to add aws4 module in order to sign requests to new AWS Bedrock connector. https://www.npmjs.com/package/aws4

ymao1

Response Ops LGTM! Left a few minor nits but overall looks great.

ymao1 · 2023-09-25T18:44:49Z

x-pack/plugins/stack_connectors/server/connector_types/bedrock/index.ts

+) => {
+  try {
+    assertURL(configObject.apiUrl);
+    urlAllowListValidator('apiUrl')(configObject, validatorServices);


is the region always a part of the apiUrl? Is it worth doing any validation that the apiUrl includes the specified region?

so i was playing around with the signature yesterday to see if I really needed to send all those arguments and discovered that i can omit method, service and region and still have successful results.

Looks like they can correctly infer the values from RequestSigner.matchHost().Therefore, I think it would be ok to omit the region from the config. WDYT?

FYI, I still need to send host, it seems the RequestSigner.createHost() does not get us what we need, here is the error:

Message: Unexpected API Error: ERR_TLS_CERT_ALTNAME_INVALID - Hostname/IP does not match certificate's altnames: Host: .us-east-1.amazonaws.com. is not in the cert's altnames: DNS:bedrock.us-east-1.amazonaws.com

I think if we can omit the region from the connector config, that'd be great. Less chance of user error entering a region that doesn't match the api url

ymao1 · 2023-09-25T19:33:04Z

x-pack/plugins/stack_connectors/server/connector_types/bedrock/render.test.ts

+const variables = { domain: 'm0zepcuuu2' };
+
+describe('Bedrock - renderParameterTemplates', () => {
+  it('should not render body on test action', () => {


in render, you're returning if the subaction doesn't equal run or test, so I'm not sure what the correct behavior is? This test passes because you're using test but no template to render in the param body

tbh I'm not sure I understand the purpose of renderParameterTemplates, I was following other connector types... in case there are mustache variables used in the test/run json form?

renderParameterTemplates is an optional function that you can register with the connector type if you need special escaping for some of the parameter values, typically while running or testing the action. This function as is is saying to only use this function for the run or test subaction and to not worry about mustache templating for any of the other subactions. Maybe you want to change it so it only renders the mustache templates during the run subaction? Then this unit test would make sense.

the test and run action are the same

x-pack/plugins/stack_connectors/common/gen_ai/types.ts

ymao1 · 2023-09-26T12:30:55Z

x-pack/plugins/stack_connectors/common/bedrock/types.ts

+export type BedrockConfig = TypeOf<typeof BedrockConfigSchema>;
+export type BedrockSecrets = TypeOf<typeof BedrockSecretsSchema>;
+export type BedrockRunActionParams = TypeOf<typeof BedrockRunActionParamsSchema>;
+export type InvokeAIActionParams = TypeOf<typeof InvokeAIActionParamsSchema>;


nit: BedrockInvokeAIActionParams and BedrockInvokeAIActionResponse for consistency?

see #166662 (comment)

ymao1 · 2023-09-26T12:32:27Z

x-pack/plugins/stack_connectors/public/connector_types/gen_ai/gen_ai.tsx

@@ -23,7 +23,7 @@ export function getConnectorType(): GenerativeAiConnector {
    selectMessage: i18n.translate('xpack.stackConnectors.components.genAi.selectMessageText', {
      defaultMessage: 'Send a request to generative AI systems.',


Should this be Send a request to Open AI systems?

I'm going to fix all the language/naming in a follow up: #166960

You can preview that PR here: stephmilovic#12

I will reopen against main once this PR is merged

x-pack/plugins/stack_connectors/server/connector_types/bedrock/bedrock.ts

x-pack/plugins/stack_connectors/server/connector_types/gen_ai/gen_ai.ts

...alerting_api_integration/security_and_spaces/group2/tests/actions/connector_types/bedrock.ts

ymao1 · 2023-09-26T13:18:32Z

x-pack/plugins/stack_connectors/server/connector_types/bedrock/bedrock.ts

+      return 'Unauthorized API Error';
+    }
+    return `API Error: ${error.response?.status} - ${error.response?.statusText}${
+      error.response?.data?.error?.message ? ` - ${error.response.data.error?.message}` : ''


When I use an invalid model, right now it just returns Error: Status code: 400. Message: API Error: 400 - Bad Request but there is a more informative error message inside error.response.data.message that would be useful The provided model identifier is invalid.

Definitely, i kind of breezed over error handling because I have another ticket to clean it up in GenAI connector so I was going to handle all of that at once if that is ok? https://github.com/elastic/security-team/issues/7373

Here is the preview of that PR: stephmilovic#13

I will reopen against main once this PR + stephmilovic#12 are merged

stephmilovic · 2023-09-27T15:27:14Z

@elasticmachine merge upstream

stephmilovic · 2023-09-27T16:04:37Z

@elasticmachine merge upstream

stephmilovic · 2023-09-27T17:32:15Z

@elasticmachine merge upstream

stephmilovic · 2023-09-27T18:48:39Z

@elasticmachine merge upstream

stephmilovic · 2023-09-27T19:27:16Z

@elasticmachine merge upstream

kibana-ci · 2023-09-27T21:57:42Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: ae9856e

Failed CI Steps

FTR Configs #59

Test Failures

[job] [logs] FTR Configs #59 / serverless observability UI landing page "before all" hook for "has quickstart badge"

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
`securitySolution`	4549	4550	+1
`stackConnectors`	239	247	+8
total			+9

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	12.8MB	12.8MB	-4.5KB
`stackConnectors`	467.0KB	485.4KB	+18.4KB
total			+13.8KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
`stackConnectors`	36.5KB	37.8KB	+1.3KB

Unknown metric groups

async chunk count

id	before	after	diff
`stackConnectors`	71	75	+4

ESLint disabled line counts

id	before	after	diff
`@kbn/elastic-assistant`	19	18	-1
`elasticAssistant`	18	10	-8
`stackConnectors`	94	98	+4
total			-5

Total ESLint disabled count

id	before	after	diff
`@kbn/elastic-assistant`	20	19	-1
`elasticAssistant`	18	10	-8
`stackConnectors`	99	103	+4
total			-5

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

stephmilovic added 3 commits September 13, 2023 14:42

wip

00cdab0

Merge branch 'main' into bedrock

f907c46

bootstrapping

45bb251

stephmilovic added release_note:enhancement WIP Work in progress Team:Threat Hunting Security Solution Threat Hunting Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.11.0 labels Sep 18, 2023

stephmilovic and others added 19 commits September 18, 2023 15:59

working

70c470f

fix

9f6c8fe

fix url

182d90e

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

75ae921

…-fix'

connector added to AI assistant

c55a851

Merge branch 'bedrock' of github.com:stephmilovic/kibana into bedrock

16902ca

fix

40ab64d

dentist

e461297

invokeAI method

4f3019c

rm log

5991de8

Merge branch 'main' into bedrock

be72b79

make langchain work

e90395e

about to combine connector selector components

e884b94

condense connector selector logic

f6aa152

undo type change

95f5a97

remove dashboard and fix types

90dfe25

fix stack_connectors tests

c5c65f7

Merge branch 'main' into bedrock

ed4773b

aws4 package instead of aws4-axios

1c61a59

stephmilovic commented Sep 20, 2023

View reviewed changes

stephmilovic added 2 commits September 20, 2023 16:27

fix type and lint

3863f09

fix test

36065f8

stephmilovic added 3 commits September 25, 2023 10:27

Merge branch 'main' into bedrock

d0549eb

fix merge

4acbbc5

add model to params form

11ae8d7

ymao1 approved these changes Sep 26, 2023

View reviewed changes

pr updates for ymao1

c6d7e94

stephmilovic mentioned this pull request Sep 26, 2023

Rename GenAI to OpenAI Preview stephmilovic/kibana#12

Closed

rm console log

30f93cc

stephmilovic mentioned this pull request Sep 26, 2023

Ai error handling stephmilovic/kibana#13

Closed

stephmilovic added 2 commits September 27, 2023 08:35

the logo is here!

ac8eb76

Merge branch 'main' into bedrock

9b872df

Merge branch 'main' into bedrock

ded3418

kibanamachine and others added 2 commits September 27, 2023 12:04

Merge branch 'main' into bedrock

bcb1c14

update screenshots

7492595

Merge branch 'main' into bedrock

2f88749

Merge branch 'main' into bedrock

7ef8a09

kibanamachine and others added 2 commits September 27, 2023 15:27

Merge branch 'main' into bedrock

5b58a3a

Merge remote-tracking branch 'upstream/main' into bedrock

ae9856e

stephmilovic merged commit bacebd2 into elastic:main Sep 27, 2023

kibanamachine added the backport:skip This commit does not require backporting label Sep 27, 2023

stephmilovic mentioned this pull request Sep 28, 2023

[Security solution] Rename Generative AI connector to OpenAI #167519

Merged

nastasha-solomon mentioned this pull request Oct 10, 2023

New Amazon Bedrock Connector elastic/security-docs#4031

Closed

This was referenced Oct 12, 2023

[OAS] Amazon Bedrock connector #168662

Merged

[DOCS] Add preconfigured AWS Bedrock connector #168663

Merged

stephmilovic mentioned this pull request Oct 17, 2023

[Security solution] Fix OpenAI token reporting #169156

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security solution] AWS Bedrock connector #166662

[Security solution] AWS Bedrock connector #166662

stephmilovic commented Sep 18, 2023 •

edited

Loading

stephmilovic Sep 20, 2023

ymao1 left a comment

ymao1 Sep 25, 2023

stephmilovic Sep 26, 2023 •

edited

Loading

ymao1 Sep 26, 2023

ymao1 Sep 25, 2023

stephmilovic Sep 26, 2023

ymao1 Sep 26, 2023

stephmilovic Sep 26, 2023

ymao1 Sep 26, 2023

stephmilovic Sep 26, 2023

ymao1 Sep 26, 2023

stephmilovic Sep 26, 2023

stephmilovic Sep 26, 2023

ymao1 Sep 26, 2023

stephmilovic Sep 26, 2023

stephmilovic Sep 26, 2023

stephmilovic commented Sep 27, 2023

stephmilovic commented Sep 27, 2023

stephmilovic commented Sep 27, 2023

stephmilovic commented Sep 27, 2023

stephmilovic commented Sep 27, 2023

kibana-ci commented Sep 27, 2023

async chunk count

ESLint disabled line counts

Total ESLint disabled count

		@@ -23,7 +23,7 @@ export function getConnectorType(): GenerativeAiConnector {
		selectMessage: i18n.translate('xpack.stackConnectors.components.genAi.selectMessageText', {
		defaultMessage: 'Send a request to generative AI systems.',

[Security solution] AWS Bedrock connector #166662

[Security solution] AWS Bedrock connector #166662

Conversation

stephmilovic commented Sep 18, 2023 • edited Loading

Summary

Create connector

Test connector

Testing for ResponseOps team

Testing for Security Solution team

Choose a reason for hiding this comment

ymao1 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stephmilovic Sep 26, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

stephmilovic commented Sep 27, 2023

stephmilovic commented Sep 27, 2023

stephmilovic commented Sep 27, 2023

stephmilovic commented Sep 27, 2023

stephmilovic commented Sep 27, 2023

kibana-ci commented Sep 27, 2023

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Module Count

Async chunks

Page load bundle

async chunk count

ESLint disabled line counts

Total ESLint disabled count

History

stephmilovic commented Sep 18, 2023 •

edited

Loading

stephmilovic Sep 26, 2023 •

edited

Loading