[FR] CI Job to Sync ES|QL Custom Fields with Prebuilt Filterlist for Telemetry

[terrancedejesus]

Repository Feature

Core Repo - (rule management, validation, testing, lib, cicd, etc.)

Problem Description

At the moment, when using ES|QL for writing detection rule queries, often we use aggregate functions, eval or pre-processing functions (grok and dissect) to create useful fields for our filters.

In this instance, those fields are not available in global alert telemetry, which relies on a static filterlist for determining what fields to ingest from the alerts.

Desired Solution

As such, we must develop a CI job that loads the ES|QL rules, reviews custom fields and adds those to the filterlist. The CI job would run on merges into main only.

Considered Alternatives

No alternatives considered. This suggestion is post conversation with Security Data Analytics team.

Additional Context

No response

[Mikaayenson]

We will need to consider how to handle the internal list in CI. We may want to kickoff a job that runs in CI in a different repo.