[Response Ops][Alerting] Adding ability to run actions for backfill rule runs #200784

ymao1 · 2024-11-19T17:27:07Z

Resolves https://github.com/elastic/response-ops-team/issues/251

Note

This PR includes some saved object schema changes that I will pull out into their own separate PR in order to perform an intermediate release. I wanted to make sure all the schema changes made sense in the overall context of the PR before opening those separate PRs.

Update: PR for intermediate release here: #203184 (Merged)

Summary

Adds ability to run actions for backfill rule runs.

Updates schedule backfill API to accept run_actions parameter to specify whether to run actions for backfill.
Schedule API accepts any action where frequency.notifyWhen === 'onActiveAlert'. If a rule has multiple actions where some are onActiveAlert and some are onThrottleInterval, the invalid actions will be stripped and a warning returned in the schedule response but valid actions will be scheduled.
Connector IDs are extracted and stored as references in the ad hoc run params saved object
Any actions that result from a backfill task run are scheduled as low priority tasks

To Verify

Create a detection rule. Make sure you have some past data that the rule can run over in order to generate actions. Make sure you add actions to the rule. For testing, I added some conditional actions so I could see actions running only on backfill runs using kibana.alert.rule.execution.type: "manual". Create actions with and without summaries.
Schedule a backfill either directly via the API or using the detection UI. Verify that actions are run for the backfill runs that generate alerts.

kibanamachine · 2024-11-21T03:34:56Z

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7453

[✅] x-pack/test/alerting_api_integration/security_and_spaces/group1/config.ts: 200/200 tests passed.

see run history

ymao1 · 2024-11-25T16:52:22Z

x-pack/plugins/actions/server/create_execute_function.ts

@@ -171,15 +177,17 @@ export function createBulkExecutionEnqueuerFunction({
          executionId: actionToExecute.executionId,
          consumer: actionToExecute.consumer,
          relatedSavedObjects: relatedSavedObjectWithRefs,
+          ...(actionToExecute.apiKeyId ? { apiKeyId: actionToExecute.apiKeyId } : {}),


Storing apiKeyId in the action_task_param saved object so that the API key invalidation task can query for any scheduled actions that are using invalidated API keys and wait before performing invalidation.

elasticmachine · 2024-11-25T21:19:51Z

Pinging @elastic/response-ops (Team:ResponseOps)

mikecote · 2024-11-27T16:53:23Z

x-pack/plugins/task_manager/server/queries/mark_available_tasks_as_claimed.ts

+          if (doc['task.priority'].size() != 0) {
+            return doc['task.priority'].value;
+          } else if (params.priority_map.containsKey(taskType)) {


Moving the conversation from #199218 (comment) here, sorry for the delay!

Either would work! The way it currently is, we would sort by priority=descending which makes more sense in my head but definitely an argument for the reverse

Hmm, is it easy to check what other queueing products like RabbitMQ, Kafka, etc use for priority order? We can see if they do asc or desc and follow suit.

For RabbitMQ: Priorities between 1 and 255 are supported, Larger numbers indicate higher priority.

For ActiveMQ: The full range of priority values (0-9) are supported by the [JDBC](https://activemq.apache.org/components/classic/documentation/jdbc-support) message store. For [KahaDB](https://activemq.apache.org/components/classic/documentation/Persistence/kahadb) three priority categories are supported, Low (< 4), Default (= 4) and High (> 4).

For Redis, priority can be implemented using sorted sets, using the score to represent task priority. For sorted sets, If B and A are two elements with a different score, then A > B if A.score is > B.score.

Kafka and SQS do not support event/message level priority. Instead they both recommend adding different topics and reading from them in priority level (read from high priority topic/queue first, then fallback to medium priority if nothing in the high priority topic/queue, etc).

Seems like in general, higher value = higher priority?

Thanks for doing the research, it sounds in line with the way we have it implemented today so we are good 👍 ❤️

jloleysens

Limited my review to the model/mappings changes. Thanks for explaining the rationale!

From what I can tell an intermediate release will be required for apiKeyId field on action_task_params 👍🏻

ymao1 · 2024-12-02T14:29:15Z

@elasticmachine merge upstream

pmuellr

LGTM

Detection rule runs backfill actions as expected, very cool. Code LGTM but marked some property setting simplifications (I think they are kosher).

One thing I'm wondering about. To figure out if the action was from a backfill, I used an original_event time value, which seemed to work. I think if I was going to use this, I'd want to have a way to know if this is a backfill, to put some kind of warning in my mustache template like

{{#isBackFill}}**Note: this was generated from a backfill**
{{/isBackFill}}

We can obviously do this later, but I was a little curious and haven't figured out where that would go, in our code ...

ymao1 · 2024-12-05T19:02:38Z

marked some property setting simplifications

@pmuellr Did you have some comments that got lost somewhere?

ymao1 · 2024-12-05T19:03:12Z

We can obviously do this later, but I was a little curious and haven't figured out where that would go, in our code ...

Good idea! I can create a follow-up issue for this.

pmuellr · 2024-12-05T19:30:05Z

Did you have some comments that got lost somewhere?

Welp dang, I did! Must have been the sporadic GH issues from yesterday. Let me see if I can find them, but they were nits anyway ...

pmuellr · 2024-12-05T19:32:36Z

x-pack/plugins/actions/server/create_execute_function.ts

@@ -171,15 +177,17 @@ export function createBulkExecutionEnqueuerFunction({
          executionId: actionToExecute.executionId,
          consumer: actionToExecute.consumer,
          relatedSavedObjects: relatedSavedObjectWithRefs,
+          ...(actionToExecute.apiKeyId ? { apiKeyId: actionToExecute.apiKeyId } : {}),


I noticed this pattern used in a few places:

...(actionToExecute.apiKeyId ? { apiKeyId: actionToExecute.apiKeyId } : {}),

I think this can be simplified to

apiKeyId: actionToExecute.apiKeyId,

I'm not aware of - but wouldn't be suprised to find - something sensitive to the property not in the object vs the property having undefined as the value.

…iate release (#203184) ## Summary This PR contains just the schema changes required to support backfill actions. This is meant for an intermediate release and then the full PR: #200784 will follow after that. --------- Co-authored-by: Elastic Machine <[email protected]>

…iate release (elastic#203184) ## Summary This PR contains just the schema changes required to support backfill actions. This is meant for an intermediate release and then the full PR: elastic#200784 will follow after that. --------- Co-authored-by: Elastic Machine <[email protected]> (cherry picked from commit b9bac16)

…iate release (elastic#203184) ## Summary This PR contains just the schema changes required to support backfill actions. This is meant for an intermediate release and then the full PR: elastic#200784 will follow after that. --------- Co-authored-by: Elastic Machine <[email protected]>

ymao1 · 2024-12-12T13:00:31Z

@elasticmachine merge upstream

…termediate release (#203184) (#203885) # Backport This will backport the following commits from `main` to `8.x`: - [[Response Ops][Alerting] Backfill actions schema changes for intermediate release (#203184)](#203184)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Ying Mao <[email protected]> Co-authored-by: Elastic Machine <[email protected]>

…ill-actions

ymao1 · 2024-12-12T20:22:38Z

One thing I'm wondering about. To figure out if the action was from a backfill, I used an original_event time value, which seemed to work. I think if I was going to use this, I'd want to have a way to know if this is a backfill, to put some kind of warning in my mustache template like
{{#isBackFill}}**Note: this was generated from a backfill**
{{/isBackFill}}
We can obviously do this later, but I was a little curious and haven't figured out where that would go, in our code ...

We have a field in our alert document: kibana.alert.rule.execution.type which can be either "manual" or "scheduled". Seems like we could do conditional actions to add 2 actions that print different messages based on whether the run was from normal execution or a backfill.

pmuellr · 2024-12-12T23:44:03Z

We have a field in our alert document: kibana.alert.rule.execution.type which can be either "manual" or "scheduled". Seems like we could do conditional actions to add 2 actions that print different messages based on whether the run was from normal execution or a backfill.

Does "manual" already have a use? Can we use a new value "backfill"?

ymao1 · 2024-12-13T01:52:31Z

Does "manual" already have a use? Can we use a new value "backfill"?

In this case manual = backfill. The detections UI calls them manual rule runs so they wanted to set the value this way so it shows up consistently in their UI.

pmuellr · 2024-12-13T21:34:56Z

Seems like we could do conditional actions to add 2 actions that print different messages based on whether the run was from normal execution or a backfill.

Ya, that should work.

I was actually thinking about context variables I guess, and not alert fields. Still think we can just do this later, and we may get some requirements for it as well ...

elasticmachine · 2025-01-02T13:07:28Z

💔 Build Failed

Buildkite Build
Commit: bc22bd3

Failed CI Steps

Check Types

History

💚 Build #260039 succeeded 0de2c31
💔 Build #259772 failed acf9b5e
💔 Build #259483 failed 7524fe9
💔 Build #257668 failed 286f472
💚 Build #256032 succeeded 8b3d7b2

cc @ymao1

ymao1 force-pushed the backfill-actions branch 8 times, most recently from 07d9365 to 8010638 Compare November 20, 2024 20:40

Adding ability to run actions for backfill rule runs

6a60ecb

ymao1 force-pushed the backfill-actions branch from bf12afa to 6a60ecb Compare November 21, 2024 00:47

Merge branch 'main' of github.com:elastic/kibana into backfill-actions

6176db2

ymao1 changed the title ~~Adding ability to run actions for backfill rule runs~~ [Response Ops][Alerting] Adding ability to run actions for backfill rule runs Nov 25, 2024

Default to true if run_actions not specified

df33d0c

ymao1 commented Nov 25, 2024

View reviewed changes

ymao1 added 3 commits November 25, 2024 12:42

Default to true if run_actions not specified

51124d3

Handling rule level notify when field

2a656bd

Cleanup

56fe8db

ymao1 self-assigned this Nov 25, 2024

ymao1 marked this pull request as ready for review November 25, 2024 21:19

ymao1 requested review from a team as code owners November 25, 2024 21:19

mikecote reviewed Nov 27, 2024

View reviewed changes

jloleysens approved these changes Nov 29, 2024

View reviewed changes

Merge branch 'main' into backfill-actions

8b3d7b2

ymao1 requested review from pmuellr and adcoelho December 2, 2024 14:29

pmuellr approved these changes Dec 5, 2024

View reviewed changes

pmuellr reviewed Dec 5, 2024

View reviewed changes

ymao1 added 2 commits December 5, 2024 16:09

Merging in main

d5ccc56

PR feedback

286f472

ymao1 mentioned this pull request Dec 6, 2024

[Response Ops][Alerting] Backfill actions schema changes for intermediate release #203184

Merged

ymao1 added 2 commits December 11, 2024 13:51

Merging in main

22bb5c7

Fixing type

7524fe9

Merge branch 'main' into backfill-actions

acf9b5e

ymao1 added 3 commits December 12, 2024 14:01

Merge branch 'main' of github.com:elastic/kibana into backfill-actions

06118d4

Fixing test

4a9ebb9

Merge branch 'backfill-actions' of github.com:ymao1/kibana into backf…

0de2c31

…ill-actions

Merge branch 'main' of github.com:elastic/kibana into backfill-actions

bc22bd3

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Response Ops][Alerting] Adding ability to run actions for backfill rule runs #200784

[Response Ops][Alerting] Adding ability to run actions for backfill rule runs #200784

ymao1 commented Nov 19, 2024 •

edited

Loading

kibanamachine commented Nov 21, 2024

ymao1 Nov 25, 2024

elasticmachine commented Nov 25, 2024

mikecote Nov 27, 2024

ymao1 Dec 2, 2024

mikecote Dec 2, 2024

jloleysens left a comment

ymao1 commented Dec 2, 2024

pmuellr left a comment

ymao1 commented Dec 5, 2024

ymao1 commented Dec 5, 2024

pmuellr commented Dec 5, 2024

pmuellr Dec 5, 2024

ymao1 commented Dec 12, 2024

ymao1 commented Dec 12, 2024

pmuellr commented Dec 12, 2024

ymao1 commented Dec 13, 2024

pmuellr commented Dec 13, 2024

elasticmachine commented Jan 2, 2025 •

edited

Loading

[Response Ops][Alerting] Adding ability to run actions for backfill rule runs #200784

Are you sure you want to change the base?

[Response Ops][Alerting] Adding ability to run actions for backfill rule runs #200784

Conversation

ymao1 commented Nov 19, 2024 • edited Loading

Note

Summary

To Verify

kibanamachine commented Nov 21, 2024

Flaky Test Runner Stats

🎉 All tests passed! - kibana-flaky-test-suite-runner#7453

ymao1 Nov 25, 2024

Choose a reason for hiding this comment

elasticmachine commented Nov 25, 2024

mikecote Nov 27, 2024

Choose a reason for hiding this comment

ymao1 Dec 2, 2024

Choose a reason for hiding this comment

mikecote Dec 2, 2024

Choose a reason for hiding this comment

jloleysens left a comment

Choose a reason for hiding this comment

ymao1 commented Dec 2, 2024

pmuellr left a comment

Choose a reason for hiding this comment

ymao1 commented Dec 5, 2024

ymao1 commented Dec 5, 2024

pmuellr commented Dec 5, 2024

pmuellr Dec 5, 2024

Choose a reason for hiding this comment

ymao1 commented Dec 12, 2024

ymao1 commented Dec 12, 2024

pmuellr commented Dec 12, 2024

ymao1 commented Dec 13, 2024

pmuellr commented Dec 13, 2024

elasticmachine commented Jan 2, 2025 • edited Loading

💔 Build Failed

Failed CI Steps

History

ymao1 commented Nov 19, 2024 •

edited

Loading

elasticmachine commented Jan 2, 2025 •

edited

Loading