Skip to content

Commit

Permalink
[Security Solution] [Attack discovery] Overrides default Attack disco…
Browse files Browse the repository at this point in the history
…very timeouts (elastic#183575)

## [Security Solution] [Attack discovery] Overrides default Attack discovery timeouts

### Summary

This PR fixes an issue where Attack discovery requests may be retried when responses from the LLM take longer than two minutes.

In LangSmith, the retry looks like the following _before_ screenshot:

#### Before

![langsmith_before](https://github.com/elastic/kibana/assets/4459398/b02f016c-c260-43f3-a6cc-1260ca8d99c2)

_Above: Before the fix, a retry, shown in LangSmith, for an LLM call > 2 minutes_

After the fix, a single pair for runs > 2 minutes are observed in LangSmith:

#### After

![langsmith_after](https://github.com/elastic/kibana/assets/4459398/864ef2d4-f845-4d62-ab30-686211aadf30)

_Above: After the fix, a single pair in LangSmith, for an LLM call > 2 minutes_

### Details

This PR overrides the following default timeouts:

1) The attack discovery route's `idleSocket` socket timeout in `x-pack/plugins/elastic_assistant/server/routes/attack_discovery/post_attack_discovery.ts`

2) The connector timeout (also in `x-pack/plugins/elastic_assistant/server/routes/attack_discovery/post_attack_discovery.ts`)

3) The chain timeout in `x-pack/plugins/security_solution/server/assistant/tools/attack_discovery/attack_discovery_tool.ts`

with the following defaults:

```typescript
const ROUTE_HANDLER_TIMEOUT = 10 * 60 * 1000; // 10 * 60 seconds = 10 minutes
const LANG_CHAIN_TIMEOUT = ROUTE_HANDLER_TIMEOUT - 10_000; // 9 minutes 50 seconds
const CONNECTOR_TIMEOUT = LANG_CHAIN_TIMEOUT - 10_000; // 9 minutes 40 seconds
```

### Desk testing

1) Verify there are ~ 100 open alerts in the last 24 hours in your testing environment

2) Navigate to Security > Attack discovery

3) Select an Azure / OpenAI connector

4) Click Generate

**Expected results**

- LangSmith displays a single pair of `LLMChain` and `AttackDiscovery` runs when the LLM responds (with the final answer) in less than 2 minutes
- LangSmith displays a single pair of `LLMChain` and `AttackDiscovery` runs when the LLM takes longer than two minutes to respond (with the final answer), as illustrated by the `before` / `after` screenshots in the description above
  • Loading branch information
andrew-goldstein authored May 16, 2024
1 parent 8b10ce2 commit 1c96c31
Show file tree
Hide file tree
Showing 4 changed files with 15 additions and 0 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ export const getAssistantToolParams = ({
alertsIndexPattern,
anonymizationFields,
esClient,
langChainTimeout,
latestReplacements,
llm,
onNewReplacements,
Expand All @@ -46,6 +47,7 @@ export const getAssistantToolParams = ({
alertsIndexPattern: string;
anonymizationFields?: AnonymizationFieldResponse[];
esClient: ElasticsearchClient;
langChainTimeout: number;
latestReplacements: Replacements;
llm: ActionsClientLlm;
onNewReplacements: (newReplacements: Replacements) => void;
Expand All @@ -61,6 +63,7 @@ export const getAssistantToolParams = ({
isEnabledKnowledgeBase: false, // not required for attack discovery
chain: undefined, // not required for attack discovery
esClient,
langChainTimeout,
llm,
modelExists: false, // not required for attack discovery
onNewReplacements,
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,10 @@ import { buildResponse } from '../../lib/build_response';
import { ElasticAssistantRequestHandlerContext } from '../../types';
import { getLlmType } from '../utils';

const ROUTE_HANDLER_TIMEOUT = 10 * 60 * 1000; // 10 * 60 seconds = 10 minutes
const LANG_CHAIN_TIMEOUT = ROUTE_HANDLER_TIMEOUT - 10_000; // 9 minutes 50 seconds
const CONNECTOR_TIMEOUT = LANG_CHAIN_TIMEOUT - 10_000; // 9 minutes 40 seconds

export const postAttackDiscoveryRoute = (
router: IRouter<ElasticAssistantRequestHandlerContext>
) => {
Expand All @@ -33,6 +37,9 @@ export const postAttackDiscoveryRoute = (
path: ATTACK_DISCOVERY,
options: {
tags: ['access:elasticAssistant'],
timeout: {
idleSocket: ROUTE_HANDLER_TIMEOUT,
},
},
})
.addVersion(
Expand Down Expand Up @@ -109,6 +116,7 @@ export const postAttackDiscoveryRoute = (
logger,
request,
temperature: 0, // zero temperature for attack discovery, because we want structured JSON output
timeout: CONNECTOR_TIMEOUT,
traceOptions,
});

Expand All @@ -117,6 +125,7 @@ export const postAttackDiscoveryRoute = (
anonymizationFields,
esClient,
latestReplacements,
langChainTimeout: LANG_CHAIN_TIMEOUT,
llm,
onNewReplacements,
request,
Expand Down
1 change: 1 addition & 0 deletions x-pack/plugins/elastic_assistant/server/types.ts
Original file line number Diff line number Diff line change
Expand Up @@ -211,6 +211,7 @@ export interface AssistantToolParams {
isEnabledKnowledgeBase: boolean;
chain?: RetrievalQAChain;
esClient: ElasticsearchClient;
langChainTimeout?: number;
llm?: ActionsClientLlm | ActionsClientChatOpenAI;
modelExists: boolean;
onNewReplacements?: (newReplacements: Replacements) => void;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ export const ATTACK_DISCOVERY_TOOL: AssistantTool = {
alertsIndexPattern,
anonymizationFields,
esClient,
langChainTimeout,
llm,
onNewReplacements,
replacements,
Expand Down Expand Up @@ -102,6 +103,7 @@ export const ATTACK_DISCOVERY_TOOL: AssistantTool = {

const result = await answerFormattingChain.call({
query: getAttackDiscoveryPrompt({ anonymizedAlerts }),
timeout: langChainTimeout,
});
const attackDiscoveries = result.records;

Expand Down

0 comments on commit 1c96c31

Please sign in to comment.