[SIEM migrations] Implement ES|QL lookups and other fixes #204960

semd · 2024-12-19T14:40:01Z

Summary

Adds support for ES|QL native LOOKUP JOIN operators for Splunk lookups.

Lookups import changes:
- Stores the lookups files as indices using lookup_<lookup_name> pattern (queries fail if the name contains -)
- Indexes the lookups content data without duplicates (supports csv and json/ndjson)
- Stores the lookup index name as the resource content that is passed to the translation agent
- Fixes bug with _lookup suffix in the names coming from Splunk: queries use the _lookup suffix, but files in the. lookup editor don't have it)
Lookups translation changes:
- Prompt for the inline_query node updated to support lookups, replacing the splunk lookup name with the new Elastic lookup index name. Placeholders for missing macros/lookups are now added in this node instead of the translate_query node.
- Prompt for ES|QL translation updated to convert LOOKUP syntax and ignore macro/lookups placeholders
Other improvements on the agent graph:
- All rule migration nodes in the graph now generate a "summary" explaining the reasoning behind each decision of the LLM, they are displayed in the comments section of each rule translation.
- The inline query node was moved inside the translation sub-graph since it's only needed there.
- Validation now is executed without placeholders, preventing it from running all the iterations without being able to fix it.
- A deterministic node was added at the end to set the translation result and ensure minimum defaults are met.
- Avoid inline_query LLM calls when a prebuilt rule matched or when the Splunk query is unsupported
- Avoid prebuilt_rule matching LLM calls when no prebuilt rule is retrieved from the semantic search.
- Avoid integration matching LLM calls when no integration is retrieved from the semantic search.
Other fixes
- Fixes bug which was setting translation FULL when we missed the integration and index pattern (logs-*). Changed to PARTIAL
- Fixes bug where the description was missing for custom translated rules, we now fallback to the splunk rule title if the description is missing
- Added summary comment for prebuilt rule matching

Screenshots

New summary comments:

Prebuilt rule matching:

matching
not matching

Query inlining summary:

Integration matching:

matching:
not matching

ES|QL translation

Needs manual translation reason:

Lookups UI:

Lookup index example:

Translation

Test data

rules.json
all_macros.json
lookups.zip (uncompress before uploading)

elasticmachine · 2024-12-19T20:21:56Z

Pinging @elastic/security-threat-hunting (Team:Threat Hunting)

elasticmachine · 2024-12-19T20:21:57Z

Pinging @elastic/security-solution (Team: SecuritySolution)

e40pud · 2024-12-19T21:21:28Z

...ns/rules/task/agent/sub_graphs/translate_rule/nodes/translation_result/translation_result.ts

+    } else if (state.validation_errors?.esql_errors) {
+      translationResult = RuleTranslationResult.PARTIAL;
+    } else if (query.match(/\[(macro|lookup):.*?\]/)) {
+      translationResult = RuleTranslationResult.PARTIAL;


I think we should not be able to get to this point because it will be caught in the previous check state.validation_errors?.esql_errors since [(macro|lookup):.*?\] like string will be a syntax error. Does not hurt to have this check though.

I changed the validation node to execute the syntax check without the placeholders, this way we only try to fix the query when there is a real syntax issue. Placeholders were causing the graph to run all iterations trying to fix a query that could not be fixed.
That's why now we check for placeholders explicitly at the end.

e40pud · 2024-12-19T21:28:13Z

...ib/siem_migrations/rules/task/agent/sub_graphs/translate_rule/nodes/validation/validation.ts

@@ -25,13 +25,14 @@ export const getValidationNode = ({ logger }: GetValidationNodeParams): GraphNod
    // We want to prevent infinite loops, so we increment the iterations counter for each validation run.
    const currentIteration = ++state.validation_errors.iterations;
    let esqlErrors: string = '';
-    if (!isEmpty(query)) {
-      const { errors, isEsqlQueryAggregating, hasMetadataOperator } = parseEsqlQuery(query);
+    const sanitizedQuery = query ? removePlaceHolders(query) : '';


won't queries like

|

[macro:something]

[lookup:something]

and similar variations, lead to an empty sanitizedQuery after removePlaceHolders(query) call? And this will suggest those queries pass ES|QL validation?

Yes, that's fine, the graph tries to fix queries that do not validate here. I don't think we should try to fix queries like these. They will be labeled as partial translation at the last translationResult node.

...rity_solution/server/lib/siem_migrations/rules/task/agent/sub_graphs/translate_rule/state.ts

...urity_solution/server/lib/siem_migrations/rules/task/retrievers/rule_migrations_retriever.ts

…ns/store_lookups

angorayc · 2024-12-30T16:07:55Z

Some minor stuff:

I found only required lookups will be displayed in the list. If users uploaded a non-required lookups, then it won't be indicated anywhere.

Screen.Recording.2024-12-30.at.15.47.57.mov

Rules with Needs manual translation status have a blank overview tab:

Screen.Recording.2024-12-30.at.16.06.10.mov

…ns/store_lookups

semd · 2025-01-02T12:35:50Z

Hey @angorayc
About your comments:

I found only required lookups will be displayed in the list. If users uploaded a non-required lookups, then it won't be indicated anywhere.

Yes, the lookups list is generated from the content uploaded previously (rules and macros). If the user provides lookups that are not in the list (they are not needed), we just ignore them, they are not even uploaded to the server. We only care about lookups that we need to perform the translations.

Screen.Recording.2024-12-30.at.15.47.57.mov
2. Rules with Needs manual translation status have a blank overview tab:

Good catch. Noted. We'll fix that in a separate PR since it's not part of this task

Thanks for the review!

P1llus

Changes looking great! I think from my perspective its nice to get it merged and anything that might need an update later it would be easier to modify it at that point, as the PR gives plenty of great improvements!

…ns/store_lookups

elasticmachine · 2025-01-08T17:52:42Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: cee4f09

Failed CI Steps

Test Failures

[job] [logs] Jest Tests #20 / StepDefinePackagePolicy default API response should display vars coming from package policy
[job] [logs] Jest Tests #20 / StepDefinePackagePolicy default API response should display vars coming from package policy

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
`securitySolution`	22.2MB	22.2MB	-61.0B

Unknown metric groups

ESLint disabled in files

id	before	after	diff
`@kbn/index-adapter`	0	1	+1

Total ESLint disabled count

id	before	after	diff
`@kbn/index-adapter`	2	3	+1

History

💚 Build #263658 succeeded 71e67d4
💔 Build #262836 failed be04b80
💚 Build #262667 succeeded f758e7d
💔 Build #262648 failed eea5487

cc @semd

kibanamachine · 2025-01-08T19:26:39Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12677626515

kibanamachine · 2025-01-08T19:31:42Z

💔 All backports failed

Status	Branch	Result
❌	8.x	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 204960

Questions ?

Please refer to the Backport tool documentation

semd added 3 commits December 19, 2024 15:38

lookus saved as indices

ad9fd90

agent changes

82b1ae7

replace index pattern first

9cf2019

semd changed the title ~~[SIEM migrations] Use native ES|QL LOOKUPS for translations~~ [SIEM migrations] Implement ES|QL lookups for translations Dec 19, 2024

semd self-assigned this Dec 19, 2024

semd added v9.0.0 Team:Threat Hunting Security Solution Threat Hunting Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. backport:version Backport to applied version labels v8.18.0 labels Dec 19, 2024

semd marked this pull request as ready for review December 19, 2024 20:21

semd requested a review from a team as a code owner December 19, 2024 20:21

e40pud reviewed Dec 19, 2024

View reviewed changes

...rity_solution/server/lib/siem_migrations/rules/task/agent/sub_graphs/translate_rule/state.ts Outdated Show resolved Hide resolved

e40pud reviewed Dec 19, 2024

View reviewed changes

...urity_solution/server/lib/siem_migrations/rules/task/retrievers/rule_migrations_retriever.ts Outdated Show resolved Hide resolved

semd added 2 commits December 20, 2024 10:24

uncomment code

85221a5

Merge remote-tracking branch 'upstream/main' into 11435/siem_migratio…

4d5a41b

…ns/store_lookups

semd added the release_note:skip Skip the PR/issue when compiling release notes label Dec 20, 2024

semd added 5 commits December 20, 2024 10:54

type fixes

f33fb45

fix test

eea5487

test fixed

f758e7d

translation agent improvements

cc221eb

Merge remote-tracking branch 'upstream/main' into 11435/siem_migratio…

be04b80

…ns/store_lookups

semd changed the title ~~[SIEM migrations] Implement ES|QL lookups for translations~~ [SIEM migrations] Implement ES|QL lookups and other fixes Dec 23, 2024

angorayc approved these changes Dec 30, 2024

View reviewed changes

fix test

ef87075

Merge remote-tracking branch 'upstream/main' into 11435/siem_migratio…

71e67d4

…ns/store_lookups

P1llus approved these changes Jan 7, 2025

View reviewed changes

semd added 2 commits January 8, 2025 15:52

Merge remote-tracking branch 'upstream/main' into 11435/siem_migratio…

85c3c07

…ns/store_lookups

resolve conflicts

cee4f09

semd enabled auto-merge (squash) January 8, 2025 16:10

semd merged commit 15a1611 into elastic:main Jan 8, 2025
8 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SIEM migrations] Implement ES|QL lookups and other fixes #204960

[SIEM migrations] Implement ES|QL lookups and other fixes #204960

semd commented Dec 19, 2024 •

edited by kibanamachine

Loading

elasticmachine commented Dec 19, 2024

elasticmachine commented Dec 19, 2024

e40pud Dec 19, 2024

semd Dec 20, 2024 •

edited

Loading

e40pud Dec 19, 2024 •

edited

Loading

semd Dec 20, 2024

angorayc commented Dec 30, 2024 •

edited

Loading

semd commented Jan 2, 2025

P1llus left a comment

elasticmachine commented Jan 8, 2025 •

edited

Loading

ESLint disabled in files

Total ESLint disabled count

kibanamachine commented Jan 8, 2025

kibanamachine commented Jan 8, 2025

[SIEM migrations] Implement ES|QL lookups and other fixes #204960

[SIEM migrations] Implement ES|QL lookups and other fixes #204960

Conversation

semd commented Dec 19, 2024 • edited by kibanamachine Loading

Summary

Screenshots

New summary comments:

Prebuilt rule matching:

Query inlining summary:

Integration matching:

ES|QL translation

Lookups UI:

Test data

elasticmachine commented Dec 19, 2024

elasticmachine commented Dec 19, 2024

e40pud Dec 19, 2024

Choose a reason for hiding this comment

semd Dec 20, 2024 • edited Loading

Choose a reason for hiding this comment

e40pud Dec 19, 2024 • edited Loading

Choose a reason for hiding this comment

semd Dec 20, 2024

Choose a reason for hiding this comment

angorayc commented Dec 30, 2024 • edited Loading

semd commented Jan 2, 2025

P1llus left a comment

Choose a reason for hiding this comment

elasticmachine commented Jan 8, 2025 • edited Loading

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

Async chunks

ESLint disabled in files

Total ESLint disabled count

History

kibanamachine commented Jan 8, 2025

kibanamachine commented Jan 8, 2025

💔 All backports failed

Manual backport

Questions ?

semd commented Dec 19, 2024 •

edited by kibanamachine

Loading

semd Dec 20, 2024 •

edited

Loading

e40pud Dec 19, 2024 •

edited

Loading

angorayc commented Dec 30, 2024 •

edited

Loading

elasticmachine commented Jan 8, 2025 •

edited

Loading