[Security Solution] Handle negative lookback in rule upgrade flyout #204317
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
maximpn
added
bug
maximpn
force-pushed
the
handle-negative-lookback-in-rule-upgrade-flyout
branch
4 times, most recently
from
3254df1 to
a8dacbc
Compare
|
Pinging @elastic/security-detections-response (Team:Detections and Resp) |
|
Pinging @elastic/security-solution (Team: SecuritySolution) |
|
Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management) |
|
@maximpn when I follow your instructions and attempt to open the upgrade flyout for the modified rule, I receive the following error, which looks to have been thrown by 
The above error occurred in ErrorBoundary: |
maximpn
force-pushed
the
handle-negative-lookback-in-rule-upgrade-flyout
branch
from
a8dacbc to
1a0d56f
Compare
|
Hi @rylnd, are you sure you pulled the latest PR changes? I double checked and it works for me locally as described in the PR description. Could you try removing the branch and pull the latest changes? |
maximpn
force-pushed
the
handle-negative-lookback-in-rule-upgrade-flyout
branch
4 times, most recently
from
ad57b91 to
54ed57a
Compare
|
@elasticmachine merge upstream |
Files by Code Ownerelastic/kibana-localization
elastic/security-detection-engine
elastic/security-detection-rule-management
elastic/security-engineering-productivity
elastic/security-solution
|
rylnd
approved these changes
Jan 17, 2025
There was a problem hiding this comment.
@maximpn thank you for taking the time to address feedback, brainstorm, and provide a well-factored solution, here. And thanks as well to @banderror for the thorough review.
Detection engine changes LGTM.
| import { calcDateMathDiff } from './calc_date_math_diff'; | ||
|
|
||
| /** | ||
| * Normalizes date math |
There was a problem hiding this comment.
I know we have the tests to demonstrate functionality, but perhaps we could expound here as well:
Suggested change
| * Normalizes date math | |
| * Normalizes date math strings by reducing time units, when possible | |
| * Example: "now-60s" -> "now-1h" | |
| * Example: "now-72s" -> "now-72s" |
|
@elasticmachine merge upstream |
💛 Build succeeded, but was flaky
Failed CI StepsMetrics [docs]Module Count
Async chunks
History
cc @maximpn |
|
Starting backport for target branches: 8.x |
💔 All backports failed
Manual backportTo create the backport manually run: Questions ?Please refer to the Backport tool documentation |
💚 All backports created successfully
Note: Successful backport PRs will be merged automatically after passing CI. Questions ?Please refer to the Backport tool documentation |
maximpn
added a commit
to maximpn/kibana
that referenced
this pull request
Jan 21, 2025
…lastic#204317) **Fixes: elastic#202715 **Fixes: elastic#204714 ## Summary This PR makes inconsistent/wrong rule's look-back duration prominent for a user. It falls back to a default 1 minute value in rule upgrade workflow. ## Details ### Negative/wrong `lookback` problem There is a difference between rule schedule value in a saved object and value represented to users - Saved object (and rule management API) has `interval`, `from` and `to` fields representing rule schedule. `interval` shows how often a rule runs in task runner. `from` and `to` stored in date math format like `now-10m` represent a date time range used to fetch source events. Task manager strives to run rules exactly every `interval` but it's not always possible due to multiple reasons like system load and various delays. To avoid any gaps to appear `from` point in time usually stands earlier than current time minus `interval`, for example `interval` is `10 minutes` and `from` is `now-12m` meaning rule will analyze events starting from 12 minutes old. `to` represents the latest point in time source events will be analyzed. - Diffable rule and UI represent rule schedule as `interval` and `lookback`. Where `interval` is the same as above and `lookback` and a time duration before current time minus `interval`. For example `interval` is `10 minutes` and lookback is `2 minutes` it means a rule will analyzing events starting with 12 minutes old until the current moment in time. Literally `interval`, `from` and `to` mean a rule runs every `interval` and analyzes events starting from `from` until `to`. Technically `from` and `to` may not have any correlation with `interval`, for example a rule may analyze one year old events. While it's reasonable for manual rule runs and gap remediation the same approach doesn't work well for usual rule schedule. Transformation between `interval`/`from`/`to` and `interval`/`lookback` works only when `to` is equal the current moment in time i.e. `now`. Rule management APIs allow to set any `from` and `to` values resulting in inconsistent rule schedule. Transformed `interval`/`lookback` value won't represent real time interval used to fetch source events for analysis. On top of that negative `lookback` value may puzzle users on the meaning of the negative sign. ### Prebuilt rules with `interval`/`from`/`to` resulting in negative `lookback` Some prebuilt rules have such `interval`, `from` and `to` field values thatnegative `lookback` is expected, for example `Multiple Okta Sessions Detected for a Single User`. It runs every `60 minutes` but has `from` field set to `now-30m` and `to` equals `now`. In the end we have `lookback` equals `to` - `from` - `interval` = `30 minutes` - `60 minutes` = `-30 minutes`. Our UI doesn't handle negative `lookback` values. It simply discards a negative sign and substitutes the rest for editing. In the case above `30 minutes` will be suggested for editing. Saving the form will result in changing `from` to `now-90m` <img width="1712" alt="image" src="https://github.com/user-attachments/assets/05519743-9562-4874-8a73-5596eeccacf2" /> ### Changes in this PR This PR mitigates rule schedule inconsistencies caused by `to` fields not using the current point in time i.e. `now`. The following was done - `DiffableRule`'s `rule_schedule` was changed to have `interval`, `from` and `to` fields instead of `interval` and `lookback` - `_perform` rule upgrade API endpoint was adapted to the new `DIffableRule`'s `rule_schedule` - Rule upgrade flyout calculates and shows `interval` and `lookback` in Diff View, readonly view and field form when `lookback` is non-negative and `to` equals `now` - Rule upgrade flyout shows `interval`, `from` and `to` in Diff View, readonly view and field form when `to` isn't equal `now` or calculated `lookback` is negative - Rule upgrade flyout shows a warning when `to` isn't equal `now` or calculated `lookback` is negative - Rule upgrade flyout's JSON Diff shows `interval` and `lookback` when `lookback` is non-negative and `to` equals `now` and shows `interval`, `from` and `to` in any other case - Rule details page shows `interval`, `from` and `to` in Diff View, readonly view and field form when `to` isn't equal `now` or calculated `lookback` is negative - `maxValue` was added to `ScheduleItemField` to have an ability to restrict input at reasonable values ## Screenshots - Rule upgrade workflow (negative look-back) <img width="2558" alt="Screenshot 2025-01-02 at 13 16 59" src="https://github.com/user-attachments/assets/b8bf727f-11ca-424f-892b-b024ba7f847a" /> <img width="2553" alt="Screenshot 2025-01-02 at 13 17 20" src="https://github.com/user-attachments/assets/9f751ea4-0ce0-4a23-a3b7-0a16494d957e" /> <img width="2558" alt="Screenshot 2025-01-02 at 13 18 24" src="https://github.com/user-attachments/assets/6908ab02-4011-4a6e-85ce-e60d5eac7993" /> - Rule upgrade workflow (positive look-back) <img width="2555" alt="Screenshot 2025-01-02 at 13 19 12" src="https://github.com/user-attachments/assets/06208210-c6cd-4842-8aef-6ade5d13bd36" /> <img width="2558" alt="Screenshot 2025-01-02 at 13 25 31" src="https://github.com/user-attachments/assets/aed38bb0-ccfb-479a-bb3b-e5442c518e63" /> - JSON view <img width="2559" alt="Screenshot 2025-01-02 at 13 31 37" src="https://github.com/user-attachments/assets/07575a81-676f-418e-8b98-48eefe11ab00" /> - Rule details page <img width="2555" alt="Screenshot 2025-01-02 at 13 13 16" src="https://github.com/user-attachments/assets/e977b752-9d50-4049-917a-af2e8e3f0dfe" /> <img width="2558" alt="Screenshot 2025-01-02 at 13 14 10" src="https://github.com/user-attachments/assets/06d6f477-5730-48ca-a240-b5e7592bf173" /> ## How to test? - Ensure the `prebuiltRulesCustomizationEnabled` feature flag is enabled - Allow internal APIs via adding `server.restrictInternalApis: false` to `kibana.dev.yaml` - Clear Elasticsearch data - Run Elasticsearch and Kibana locally (do not open Kibana in a web browser) - Install an outdated version of the `security_detection_engine` Fleet package ```bash curl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 2023-10-31" -d '{"force":true}' http://localhost:5601/kbn/api/fleet/epm/packages/security_detection_engine/8.14.1 ``` - Install prebuilt rules ```bash curl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 1" -d '{"mode":"ALL_RULES"}' http://localhost:5601/kbn/internal/detection_engine/prebuilt_rules/installation/_perform ``` - Set "inconsistent" rule schedule for `Suspicious File Creation via Kworker` rule by running a query below ```bash curl -X PATCH --user elastic:changeme -H "Content-Type: application/json" -H "elastic-api-version: 2023-10-31" -H "kbn-xsrf: 123" -d '{"rule_id":"ae343298-97bc-47bc-9ea2-5f2ad831c16e","interval":"10m","from":"now-5m","to":"now-2m"}' http://localhost:5601/kbn/api/detection_engine/rules ``` - Open rule upgrade flyout for `Suspicious File Creation via Kworker` rule --------- Co-authored-by: Elastic Machine <[email protected]> (cherry picked from commit 30bb71a) # Conflicts: # .github/CODEOWNERS
maximpn
added a commit
that referenced
this pull request
Jan 21, 2025
…yout (#204317) (#207302) # Backport This will backport the following commits from `main` to `8.x`: - [[Security Solution] Handle negative lookback in rule upgrade flyout (#204317)](#204317) <!--- Backport version: 9.6.4 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sorenlouv/backport) <!--BACKPORT [{"author":{"name":"Maxim Palenov","email":"[email protected]"},"sourceCommit":{"committedDate":"2025-01-20T13:41:23Z","message":"[Security Solution] Handle negative lookback in rule upgrade flyout (#204317)\n\n**Fixes: https://github.com/elastic/kibana/issues/202715**\r\n**Fixes: https://github.com/elastic/kibana/issues/204714**\r\n\r\n## Summary\r\n\r\nThis PR makes inconsistent/wrong rule's look-back duration prominent for\r\na user. It falls back to a default 1 minute value in rule upgrade\r\nworkflow.\r\n\r\n## Details\r\n\r\n### Negative/wrong `lookback` problem\r\n\r\nThere is a difference between rule schedule value in a saved object and\r\nvalue represented to users\r\n\r\n- Saved object (and rule management API) has `interval`, `from` and `to`\r\nfields representing rule schedule. `interval` shows how often a rule\r\nruns in task runner. `from` and `to` stored in date math format like\r\n`now-10m` represent a date time range used to fetch source events. Task\r\nmanager strives to run rules exactly every `interval` but it's not\r\nalways possible due to multiple reasons like system load and various\r\ndelays. To avoid any gaps to appear `from` point in time usually stands\r\nearlier than current time minus `interval`, for example `interval` is\r\n`10 minutes` and `from` is `now-12m` meaning rule will analyze events\r\nstarting from 12 minutes old. `to` represents the latest point in time\r\nsource events will be analyzed.\r\n- Diffable rule and UI represent rule schedule as `interval` and\r\n`lookback`. Where `interval` is the same as above and `lookback` and a\r\ntime duration before current time minus `interval`. For example\r\n`interval` is `10 minutes` and lookback is `2 minutes` it means a rule\r\nwill analyzing events starting with 12 minutes old until the current\r\nmoment in time.\r\n\r\nLiterally `interval`, `from` and `to` mean a rule runs every `interval`\r\nand analyzes events starting from `from` until `to`. Technically `from`\r\nand `to` may not have any correlation with `interval`, for example a\r\nrule may analyze one year old events. While it's reasonable for manual\r\nrule runs and gap remediation the same approach doesn't work well for\r\nusual rule schedule. Transformation between `interval`/`from`/`to` and\r\n`interval`/`lookback` works only when `to` is equal the current moment\r\nin time i.e. `now`.\r\n\r\nRule management APIs allow to set any `from` and `to` values resulting\r\nin inconsistent rule schedule. Transformed `interval`/`lookback` value\r\nwon't represent real time interval used to fetch source events for\r\nanalysis. On top of that negative `lookback` value may puzzle users on\r\nthe meaning of the negative sign.\r\n\r\n### Prebuilt rules with `interval`/`from`/`to` resulting in negative\r\n`lookback`\r\n\r\nSome prebuilt rules have such `interval`, `from` and `to` field values\r\nthatnegative `lookback` is expected, for example `Multiple Okta Sessions\r\nDetected for a Single User`. It runs every `60 minutes` but has `from`\r\nfield set to `now-30m` and `to` equals `now`. In the end we have\r\n`lookback` equals `to` - `from` - `interval` = `30 minutes` - `60\r\nminutes` = `-30 minutes`.\r\n\r\nOur UI doesn't handle negative `lookback` values. It simply discards a\r\nnegative sign and substitutes the rest for editing. In the case above\r\n`30 minutes` will be suggested for editing. Saving the form will result\r\nin changing `from` to `now-90m`\r\n\r\n<img width=\"1712\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/05519743-9562-4874-8a73-5596eeccacf2\"\r\n/>\r\n\r\n### Changes in this PR\r\n\r\nThis PR mitigates rule schedule inconsistencies caused by `to` fields\r\nnot using the current point in time i.e. `now`. The following was done\r\n\r\n- `DiffableRule`'s `rule_schedule` was changed to have `interval`,\r\n`from` and `to` fields instead of `interval` and `lookback`\r\n- `_perform` rule upgrade API endpoint was adapted to the new\r\n`DIffableRule`'s `rule_schedule`\r\n- Rule upgrade flyout calculates and shows `interval` and `lookback` in\r\nDiff View, readonly view and field form when `lookback` is non-negative\r\nand `to` equals `now`\r\n- Rule upgrade flyout shows `interval`, `from` and `to` in Diff View,\r\nreadonly view and field form when `to` isn't equal `now` or calculated\r\n`lookback` is negative\r\n- Rule upgrade flyout shows a warning when `to` isn't equal `now` or\r\ncalculated `lookback` is negative\r\n- Rule upgrade flyout's JSON Diff shows `interval` and `lookback` when\r\n`lookback` is non-negative and `to` equals `now` and shows `interval`,\r\n`from` and `to` in any other case\r\n- Rule details page shows `interval`, `from` and `to` in Diff View,\r\nreadonly view and field form when `to` isn't equal `now` or calculated\r\n`lookback` is negative\r\n- `maxValue` was added to `ScheduleItemField` to have an ability to\r\nrestrict input at reasonable values\r\n\r\n## Screenshots\r\n\r\n- Rule upgrade workflow (negative look-back)\r\n\r\n<img width=\"2558\" alt=\"Screenshot 2025-01-02 at 13 16 59\"\r\nsrc=\"https://github.com/user-attachments/assets/b8bf727f-11ca-424f-892b-b024ba7f847a\"\r\n/>\r\n\r\n<img width=\"2553\" alt=\"Screenshot 2025-01-02 at 13 17 20\"\r\nsrc=\"https://github.com/user-attachments/assets/9f751ea4-0ce0-4a23-a3b7-0a16494d957e\"\r\n/>\r\n\r\n<img width=\"2558\" alt=\"Screenshot 2025-01-02 at 13 18 24\"\r\nsrc=\"https://github.com/user-attachments/assets/6908ab02-4011-4a6e-85ce-e60d5eac7993\"\r\n/>\r\n\r\n- Rule upgrade workflow (positive look-back)\r\n\r\n<img width=\"2555\" alt=\"Screenshot 2025-01-02 at 13 19 12\"\r\nsrc=\"https://github.com/user-attachments/assets/06208210-c6cd-4842-8aef-6ade5d13bd36\"\r\n/>\r\n\r\n<img width=\"2558\" alt=\"Screenshot 2025-01-02 at 13 25 31\"\r\nsrc=\"https://github.com/user-attachments/assets/aed38bb0-ccfb-479a-bb3b-e5442c518e63\"\r\n/>\r\n\r\n- JSON view\r\n\r\n<img width=\"2559\" alt=\"Screenshot 2025-01-02 at 13 31 37\"\r\nsrc=\"https://github.com/user-attachments/assets/07575a81-676f-418e-8b98-48eefe11ab00\"\r\n/>\r\n\r\n- Rule details page\r\n\r\n<img width=\"2555\" alt=\"Screenshot 2025-01-02 at 13 13 16\"\r\nsrc=\"https://github.com/user-attachments/assets/e977b752-9d50-4049-917a-af2e8e3f0dfe\"\r\n/>\r\n\r\n<img width=\"2558\" alt=\"Screenshot 2025-01-02 at 13 14 10\"\r\nsrc=\"https://github.com/user-attachments/assets/06d6f477-5730-48ca-a240-b5e7592bf173\"\r\n/>\r\n\r\n## How to test?\r\n\r\n- Ensure the `prebuiltRulesCustomizationEnabled` feature flag is enabled\r\n- Allow internal APIs via adding `server.restrictInternalApis: false` to\r\n`kibana.dev.yaml`\r\n- Clear Elasticsearch data\r\n- Run Elasticsearch and Kibana locally (do not open Kibana in a web\r\nbrowser)\r\n- Install an outdated version of the `security_detection_engine` Fleet\r\npackage\r\n```bash\r\ncurl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H \"elastic-api-version: 2023-10-31\" -d '{\"force\":true}' http://localhost:5601/kbn/api/fleet/epm/packages/security_detection_engine/8.14.1\r\n```\r\n\r\n- Install prebuilt rules\r\n```bash\r\ncurl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H \"elastic-api-version: 1\" -d '{\"mode\":\"ALL_RULES\"}' http://localhost:5601/kbn/internal/detection_engine/prebuilt_rules/installation/_perform\r\n```\r\n\r\n- Set \"inconsistent\" rule schedule for `Suspicious File Creation via\r\nKworker` rule by running a query below\r\n```bash\r\ncurl -X PATCH --user elastic:changeme -H \"Content-Type: application/json\" -H \"elastic-api-version: 2023-10-31\" -H \"kbn-xsrf: 123\" -d '{\"rule_id\":\"ae343298-97bc-47bc-9ea2-5f2ad831c16e\",\"interval\":\"10m\",\"from\":\"now-5m\",\"to\":\"now-2m\"}' http://localhost:5601/kbn/api/detection_engine/rules\r\n```\r\n\r\n- Open rule upgrade flyout for `Suspicious File Creation via Kworker`\r\nrule\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"30bb71a516cf0e8e83caab99f9119057a3b1bc82","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["bug","release_note:skip","impact:high","v9.0.0","Team:Detections and Resp","Team: SecuritySolution","Team:Detection Rule Management","Feature:Prebuilt Detection Rules","backport:version","v8.18.0"],"title":"[Security Solution] Handle negative lookback in rule upgrade flyout","number":204317,"url":"https://github.com/elastic/kibana/pull/204317","mergeCommit":{"message":"[Security Solution] Handle negative lookback in rule upgrade flyout (#204317)\n\n**Fixes: https://github.com/elastic/kibana/issues/202715**\r\n**Fixes: https://github.com/elastic/kibana/issues/204714**\r\n\r\n## Summary\r\n\r\nThis PR makes inconsistent/wrong rule's look-back duration prominent for\r\na user. It falls back to a default 1 minute value in rule upgrade\r\nworkflow.\r\n\r\n## Details\r\n\r\n### Negative/wrong `lookback` problem\r\n\r\nThere is a difference between rule schedule value in a saved object and\r\nvalue represented to users\r\n\r\n- Saved object (and rule management API) has `interval`, `from` and `to`\r\nfields representing rule schedule. `interval` shows how often a rule\r\nruns in task runner. `from` and `to` stored in date math format like\r\n`now-10m` represent a date time range used to fetch source events. Task\r\nmanager strives to run rules exactly every `interval` but it's not\r\nalways possible due to multiple reasons like system load and various\r\ndelays. To avoid any gaps to appear `from` point in time usually stands\r\nearlier than current time minus `interval`, for example `interval` is\r\n`10 minutes` and `from` is `now-12m` meaning rule will analyze events\r\nstarting from 12 minutes old. `to` represents the latest point in time\r\nsource events will be analyzed.\r\n- Diffable rule and UI represent rule schedule as `interval` and\r\n`lookback`. Where `interval` is the same as above and `lookback` and a\r\ntime duration before current time minus `interval`. For example\r\n`interval` is `10 minutes` and lookback is `2 minutes` it means a rule\r\nwill analyzing events starting with 12 minutes old until the current\r\nmoment in time.\r\n\r\nLiterally `interval`, `from` and `to` mean a rule runs every `interval`\r\nand analyzes events starting from `from` until `to`. Technically `from`\r\nand `to` may not have any correlation with `interval`, for example a\r\nrule may analyze one year old events. While it's reasonable for manual\r\nrule runs and gap remediation the same approach doesn't work well for\r\nusual rule schedule. Transformation between `interval`/`from`/`to` and\r\n`interval`/`lookback` works only when `to` is equal the current moment\r\nin time i.e. `now`.\r\n\r\nRule management APIs allow to set any `from` and `to` values resulting\r\nin inconsistent rule schedule. Transformed `interval`/`lookback` value\r\nwon't represent real time interval used to fetch source events for\r\nanalysis. On top of that negative `lookback` value may puzzle users on\r\nthe meaning of the negative sign.\r\n\r\n### Prebuilt rules with `interval`/`from`/`to` resulting in negative\r\n`lookback`\r\n\r\nSome prebuilt rules have such `interval`, `from` and `to` field values\r\nthatnegative `lookback` is expected, for example `Multiple Okta Sessions\r\nDetected for a Single User`. It runs every `60 minutes` but has `from`\r\nfield set to `now-30m` and `to` equals `now`. In the end we have\r\n`lookback` equals `to` - `from` - `interval` = `30 minutes` - `60\r\nminutes` = `-30 minutes`.\r\n\r\nOur UI doesn't handle negative `lookback` values. It simply discards a\r\nnegative sign and substitutes the rest for editing. In the case above\r\n`30 minutes` will be suggested for editing. Saving the form will result\r\nin changing `from` to `now-90m`\r\n\r\n<img width=\"1712\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/05519743-9562-4874-8a73-5596eeccacf2\"\r\n/>\r\n\r\n### Changes in this PR\r\n\r\nThis PR mitigates rule schedule inconsistencies caused by `to` fields\r\nnot using the current point in time i.e. `now`. The following was done\r\n\r\n- `DiffableRule`'s `rule_schedule` was changed to have `interval`,\r\n`from` and `to` fields instead of `interval` and `lookback`\r\n- `_perform` rule upgrade API endpoint was adapted to the new\r\n`DIffableRule`'s `rule_schedule`\r\n- Rule upgrade flyout calculates and shows `interval` and `lookback` in\r\nDiff View, readonly view and field form when `lookback` is non-negative\r\nand `to` equals `now`\r\n- Rule upgrade flyout shows `interval`, `from` and `to` in Diff View,\r\nreadonly view and field form when `to` isn't equal `now` or calculated\r\n`lookback` is negative\r\n- Rule upgrade flyout shows a warning when `to` isn't equal `now` or\r\ncalculated `lookback` is negative\r\n- Rule upgrade flyout's JSON Diff shows `interval` and `lookback` when\r\n`lookback` is non-negative and `to` equals `now` and shows `interval`,\r\n`from` and `to` in any other case\r\n- Rule details page shows `interval`, `from` and `to` in Diff View,\r\nreadonly view and field form when `to` isn't equal `now` or calculated\r\n`lookback` is negative\r\n- `maxValue` was added to `ScheduleItemField` to have an ability to\r\nrestrict input at reasonable values\r\n\r\n## Screenshots\r\n\r\n- Rule upgrade workflow (negative look-back)\r\n\r\n<img width=\"2558\" alt=\"Screenshot 2025-01-02 at 13 16 59\"\r\nsrc=\"https://github.com/user-attachments/assets/b8bf727f-11ca-424f-892b-b024ba7f847a\"\r\n/>\r\n\r\n<img width=\"2553\" alt=\"Screenshot 2025-01-02 at 13 17 20\"\r\nsrc=\"https://github.com/user-attachments/assets/9f751ea4-0ce0-4a23-a3b7-0a16494d957e\"\r\n/>\r\n\r\n<img width=\"2558\" alt=\"Screenshot 2025-01-02 at 13 18 24\"\r\nsrc=\"https://github.com/user-attachments/assets/6908ab02-4011-4a6e-85ce-e60d5eac7993\"\r\n/>\r\n\r\n- Rule upgrade workflow (positive look-back)\r\n\r\n<img width=\"2555\" alt=\"Screenshot 2025-01-02 at 13 19 12\"\r\nsrc=\"https://github.com/user-attachments/assets/06208210-c6cd-4842-8aef-6ade5d13bd36\"\r\n/>\r\n\r\n<img width=\"2558\" alt=\"Screenshot 2025-01-02 at 13 25 31\"\r\nsrc=\"https://github.com/user-attachments/assets/aed38bb0-ccfb-479a-bb3b-e5442c518e63\"\r\n/>\r\n\r\n- JSON view\r\n\r\n<img width=\"2559\" alt=\"Screenshot 2025-01-02 at 13 31 37\"\r\nsrc=\"https://github.com/user-attachments/assets/07575a81-676f-418e-8b98-48eefe11ab00\"\r\n/>\r\n\r\n- Rule details page\r\n\r\n<img width=\"2555\" alt=\"Screenshot 2025-01-02 at 13 13 16\"\r\nsrc=\"https://github.com/user-attachments/assets/e977b752-9d50-4049-917a-af2e8e3f0dfe\"\r\n/>\r\n\r\n<img width=\"2558\" alt=\"Screenshot 2025-01-02 at 13 14 10\"\r\nsrc=\"https://github.com/user-attachments/assets/06d6f477-5730-48ca-a240-b5e7592bf173\"\r\n/>\r\n\r\n## How to test?\r\n\r\n- Ensure the `prebuiltRulesCustomizationEnabled` feature flag is enabled\r\n- Allow internal APIs via adding `server.restrictInternalApis: false` to\r\n`kibana.dev.yaml`\r\n- Clear Elasticsearch data\r\n- Run Elasticsearch and Kibana locally (do not open Kibana in a web\r\nbrowser)\r\n- Install an outdated version of the `security_detection_engine` Fleet\r\npackage\r\n```bash\r\ncurl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H \"elastic-api-version: 2023-10-31\" -d '{\"force\":true}' http://localhost:5601/kbn/api/fleet/epm/packages/security_detection_engine/8.14.1\r\n```\r\n\r\n- Install prebuilt rules\r\n```bash\r\ncurl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H \"elastic-api-version: 1\" -d '{\"mode\":\"ALL_RULES\"}' http://localhost:5601/kbn/internal/detection_engine/prebuilt_rules/installation/_perform\r\n```\r\n\r\n- Set \"inconsistent\" rule schedule for `Suspicious File Creation via\r\nKworker` rule by running a query below\r\n```bash\r\ncurl -X PATCH --user elastic:changeme -H \"Content-Type: application/json\" -H \"elastic-api-version: 2023-10-31\" -H \"kbn-xsrf: 123\" -d '{\"rule_id\":\"ae343298-97bc-47bc-9ea2-5f2ad831c16e\",\"interval\":\"10m\",\"from\":\"now-5m\",\"to\":\"now-2m\"}' http://localhost:5601/kbn/api/detection_engine/rules\r\n```\r\n\r\n- Open rule upgrade flyout for `Suspicious File Creation via Kworker`\r\nrule\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"30bb71a516cf0e8e83caab99f9119057a3b1bc82"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/204317","number":204317,"mergeCommit":{"message":"[Security Solution] Handle negative lookback in rule upgrade flyout (#204317)\n\n**Fixes: https://github.com/elastic/kibana/issues/202715**\r\n**Fixes: https://github.com/elastic/kibana/issues/204714**\r\n\r\n## Summary\r\n\r\nThis PR makes inconsistent/wrong rule's look-back duration prominent for\r\na user. It falls back to a default 1 minute value in rule upgrade\r\nworkflow.\r\n\r\n## Details\r\n\r\n### Negative/wrong `lookback` problem\r\n\r\nThere is a difference between rule schedule value in a saved object and\r\nvalue represented to users\r\n\r\n- Saved object (and rule management API) has `interval`, `from` and `to`\r\nfields representing rule schedule. `interval` shows how often a rule\r\nruns in task runner. `from` and `to` stored in date math format like\r\n`now-10m` represent a date time range used to fetch source events. Task\r\nmanager strives to run rules exactly every `interval` but it's not\r\nalways possible due to multiple reasons like system load and various\r\ndelays. To avoid any gaps to appear `from` point in time usually stands\r\nearlier than current time minus `interval`, for example `interval` is\r\n`10 minutes` and `from` is `now-12m` meaning rule will analyze events\r\nstarting from 12 minutes old. `to` represents the latest point in time\r\nsource events will be analyzed.\r\n- Diffable rule and UI represent rule schedule as `interval` and\r\n`lookback`. Where `interval` is the same as above and `lookback` and a\r\ntime duration before current time minus `interval`. For example\r\n`interval` is `10 minutes` and lookback is `2 minutes` it means a rule\r\nwill analyzing events starting with 12 minutes old until the current\r\nmoment in time.\r\n\r\nLiterally `interval`, `from` and `to` mean a rule runs every `interval`\r\nand analyzes events starting from `from` until `to`. Technically `from`\r\nand `to` may not have any correlation with `interval`, for example a\r\nrule may analyze one year old events. While it's reasonable for manual\r\nrule runs and gap remediation the same approach doesn't work well for\r\nusual rule schedule. Transformation between `interval`/`from`/`to` and\r\n`interval`/`lookback` works only when `to` is equal the current moment\r\nin time i.e. `now`.\r\n\r\nRule management APIs allow to set any `from` and `to` values resulting\r\nin inconsistent rule schedule. Transformed `interval`/`lookback` value\r\nwon't represent real time interval used to fetch source events for\r\nanalysis. On top of that negative `lookback` value may puzzle users on\r\nthe meaning of the negative sign.\r\n\r\n### Prebuilt rules with `interval`/`from`/`to` resulting in negative\r\n`lookback`\r\n\r\nSome prebuilt rules have such `interval`, `from` and `to` field values\r\nthatnegative `lookback` is expected, for example `Multiple Okta Sessions\r\nDetected for a Single User`. It runs every `60 minutes` but has `from`\r\nfield set to `now-30m` and `to` equals `now`. In the end we have\r\n`lookback` equals `to` - `from` - `interval` = `30 minutes` - `60\r\nminutes` = `-30 minutes`.\r\n\r\nOur UI doesn't handle negative `lookback` values. It simply discards a\r\nnegative sign and substitutes the rest for editing. In the case above\r\n`30 minutes` will be suggested for editing. Saving the form will result\r\nin changing `from` to `now-90m`\r\n\r\n<img width=\"1712\" alt=\"image\"\r\nsrc=\"https://github.com/user-attachments/assets/05519743-9562-4874-8a73-5596eeccacf2\"\r\n/>\r\n\r\n### Changes in this PR\r\n\r\nThis PR mitigates rule schedule inconsistencies caused by `to` fields\r\nnot using the current point in time i.e. `now`. The following was done\r\n\r\n- `DiffableRule`'s `rule_schedule` was changed to have `interval`,\r\n`from` and `to` fields instead of `interval` and `lookback`\r\n- `_perform` rule upgrade API endpoint was adapted to the new\r\n`DIffableRule`'s `rule_schedule`\r\n- Rule upgrade flyout calculates and shows `interval` and `lookback` in\r\nDiff View, readonly view and field form when `lookback` is non-negative\r\nand `to` equals `now`\r\n- Rule upgrade flyout shows `interval`, `from` and `to` in Diff View,\r\nreadonly view and field form when `to` isn't equal `now` or calculated\r\n`lookback` is negative\r\n- Rule upgrade flyout shows a warning when `to` isn't equal `now` or\r\ncalculated `lookback` is negative\r\n- Rule upgrade flyout's JSON Diff shows `interval` and `lookback` when\r\n`lookback` is non-negative and `to` equals `now` and shows `interval`,\r\n`from` and `to` in any other case\r\n- Rule details page shows `interval`, `from` and `to` in Diff View,\r\nreadonly view and field form when `to` isn't equal `now` or calculated\r\n`lookback` is negative\r\n- `maxValue` was added to `ScheduleItemField` to have an ability to\r\nrestrict input at reasonable values\r\n\r\n## Screenshots\r\n\r\n- Rule upgrade workflow (negative look-back)\r\n\r\n<img width=\"2558\" alt=\"Screenshot 2025-01-02 at 13 16 59\"\r\nsrc=\"https://github.com/user-attachments/assets/b8bf727f-11ca-424f-892b-b024ba7f847a\"\r\n/>\r\n\r\n<img width=\"2553\" alt=\"Screenshot 2025-01-02 at 13 17 20\"\r\nsrc=\"https://github.com/user-attachments/assets/9f751ea4-0ce0-4a23-a3b7-0a16494d957e\"\r\n/>\r\n\r\n<img width=\"2558\" alt=\"Screenshot 2025-01-02 at 13 18 24\"\r\nsrc=\"https://github.com/user-attachments/assets/6908ab02-4011-4a6e-85ce-e60d5eac7993\"\r\n/>\r\n\r\n- Rule upgrade workflow (positive look-back)\r\n\r\n<img width=\"2555\" alt=\"Screenshot 2025-01-02 at 13 19 12\"\r\nsrc=\"https://github.com/user-attachments/assets/06208210-c6cd-4842-8aef-6ade5d13bd36\"\r\n/>\r\n\r\n<img width=\"2558\" alt=\"Screenshot 2025-01-02 at 13 25 31\"\r\nsrc=\"https://github.com/user-attachments/assets/aed38bb0-ccfb-479a-bb3b-e5442c518e63\"\r\n/>\r\n\r\n- JSON view\r\n\r\n<img width=\"2559\" alt=\"Screenshot 2025-01-02 at 13 31 37\"\r\nsrc=\"https://github.com/user-attachments/assets/07575a81-676f-418e-8b98-48eefe11ab00\"\r\n/>\r\n\r\n- Rule details page\r\n\r\n<img width=\"2555\" alt=\"Screenshot 2025-01-02 at 13 13 16\"\r\nsrc=\"https://github.com/user-attachments/assets/e977b752-9d50-4049-917a-af2e8e3f0dfe\"\r\n/>\r\n\r\n<img width=\"2558\" alt=\"Screenshot 2025-01-02 at 13 14 10\"\r\nsrc=\"https://github.com/user-attachments/assets/06d6f477-5730-48ca-a240-b5e7592bf173\"\r\n/>\r\n\r\n## How to test?\r\n\r\n- Ensure the `prebuiltRulesCustomizationEnabled` feature flag is enabled\r\n- Allow internal APIs via adding `server.restrictInternalApis: false` to\r\n`kibana.dev.yaml`\r\n- Clear Elasticsearch data\r\n- Run Elasticsearch and Kibana locally (do not open Kibana in a web\r\nbrowser)\r\n- Install an outdated version of the `security_detection_engine` Fleet\r\npackage\r\n```bash\r\ncurl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H \"elastic-api-version: 2023-10-31\" -d '{\"force\":true}' http://localhost:5601/kbn/api/fleet/epm/packages/security_detection_engine/8.14.1\r\n```\r\n\r\n- Install prebuilt rules\r\n```bash\r\ncurl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H \"elastic-api-version: 1\" -d '{\"mode\":\"ALL_RULES\"}' http://localhost:5601/kbn/internal/detection_engine/prebuilt_rules/installation/_perform\r\n```\r\n\r\n- Set \"inconsistent\" rule schedule for `Suspicious File Creation via\r\nKworker` rule by running a query below\r\n```bash\r\ncurl -X PATCH --user elastic:changeme -H \"Content-Type: application/json\" -H \"elastic-api-version: 2023-10-31\" -H \"kbn-xsrf: 123\" -d '{\"rule_id\":\"ae343298-97bc-47bc-9ea2-5f2ad831c16e\",\"interval\":\"10m\",\"from\":\"now-5m\",\"to\":\"now-2m\"}' http://localhost:5601/kbn/api/detection_engine/rules\r\n```\r\n\r\n- Open rule upgrade flyout for `Suspicious File Creation via Kworker`\r\nrule\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine <[email protected]>","sha":"30bb71a516cf0e8e83caab99f9119057a3b1bc82"}},{"branch":"8.x","label":"v8.18.0","branchLabelMappingKey":"^v8.18.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> --------- Co-authored-by: kibanamachine <[email protected]>
cqliu1
pushed a commit
to cqliu1/kibana
that referenced
this pull request
Jan 21, 2025
…lastic#204317) **Fixes: elastic#202715 **Fixes: elastic#204714 ## Summary This PR makes inconsistent/wrong rule's look-back duration prominent for a user. It falls back to a default 1 minute value in rule upgrade workflow. ## Details ### Negative/wrong `lookback` problem There is a difference between rule schedule value in a saved object and value represented to users - Saved object (and rule management API) has `interval`, `from` and `to` fields representing rule schedule. `interval` shows how often a rule runs in task runner. `from` and `to` stored in date math format like `now-10m` represent a date time range used to fetch source events. Task manager strives to run rules exactly every `interval` but it's not always possible due to multiple reasons like system load and various delays. To avoid any gaps to appear `from` point in time usually stands earlier than current time minus `interval`, for example `interval` is `10 minutes` and `from` is `now-12m` meaning rule will analyze events starting from 12 minutes old. `to` represents the latest point in time source events will be analyzed. - Diffable rule and UI represent rule schedule as `interval` and `lookback`. Where `interval` is the same as above and `lookback` and a time duration before current time minus `interval`. For example `interval` is `10 minutes` and lookback is `2 minutes` it means a rule will analyzing events starting with 12 minutes old until the current moment in time. Literally `interval`, `from` and `to` mean a rule runs every `interval` and analyzes events starting from `from` until `to`. Technically `from` and `to` may not have any correlation with `interval`, for example a rule may analyze one year old events. While it's reasonable for manual rule runs and gap remediation the same approach doesn't work well for usual rule schedule. Transformation between `interval`/`from`/`to` and `interval`/`lookback` works only when `to` is equal the current moment in time i.e. `now`. Rule management APIs allow to set any `from` and `to` values resulting in inconsistent rule schedule. Transformed `interval`/`lookback` value won't represent real time interval used to fetch source events for analysis. On top of that negative `lookback` value may puzzle users on the meaning of the negative sign. ### Prebuilt rules with `interval`/`from`/`to` resulting in negative `lookback` Some prebuilt rules have such `interval`, `from` and `to` field values thatnegative `lookback` is expected, for example `Multiple Okta Sessions Detected for a Single User`. It runs every `60 minutes` but has `from` field set to `now-30m` and `to` equals `now`. In the end we have `lookback` equals `to` - `from` - `interval` = `30 minutes` - `60 minutes` = `-30 minutes`. Our UI doesn't handle negative `lookback` values. It simply discards a negative sign and substitutes the rest for editing. In the case above `30 minutes` will be suggested for editing. Saving the form will result in changing `from` to `now-90m` <img width="1712" alt="image" src="https://github.com/user-attachments/assets/05519743-9562-4874-8a73-5596eeccacf2" /> ### Changes in this PR This PR mitigates rule schedule inconsistencies caused by `to` fields not using the current point in time i.e. `now`. The following was done - `DiffableRule`'s `rule_schedule` was changed to have `interval`, `from` and `to` fields instead of `interval` and `lookback` - `_perform` rule upgrade API endpoint was adapted to the new `DIffableRule`'s `rule_schedule` - Rule upgrade flyout calculates and shows `interval` and `lookback` in Diff View, readonly view and field form when `lookback` is non-negative and `to` equals `now` - Rule upgrade flyout shows `interval`, `from` and `to` in Diff View, readonly view and field form when `to` isn't equal `now` or calculated `lookback` is negative - Rule upgrade flyout shows a warning when `to` isn't equal `now` or calculated `lookback` is negative - Rule upgrade flyout's JSON Diff shows `interval` and `lookback` when `lookback` is non-negative and `to` equals `now` and shows `interval`, `from` and `to` in any other case - Rule details page shows `interval`, `from` and `to` in Diff View, readonly view and field form when `to` isn't equal `now` or calculated `lookback` is negative - `maxValue` was added to `ScheduleItemField` to have an ability to restrict input at reasonable values ## Screenshots - Rule upgrade workflow (negative look-back) <img width="2558" alt="Screenshot 2025-01-02 at 13 16 59" src="https://github.com/user-attachments/assets/b8bf727f-11ca-424f-892b-b024ba7f847a" /> <img width="2553" alt="Screenshot 2025-01-02 at 13 17 20" src="https://github.com/user-attachments/assets/9f751ea4-0ce0-4a23-a3b7-0a16494d957e" /> <img width="2558" alt="Screenshot 2025-01-02 at 13 18 24" src="https://github.com/user-attachments/assets/6908ab02-4011-4a6e-85ce-e60d5eac7993" /> - Rule upgrade workflow (positive look-back) <img width="2555" alt="Screenshot 2025-01-02 at 13 19 12" src="https://github.com/user-attachments/assets/06208210-c6cd-4842-8aef-6ade5d13bd36" /> <img width="2558" alt="Screenshot 2025-01-02 at 13 25 31" src="https://github.com/user-attachments/assets/aed38bb0-ccfb-479a-bb3b-e5442c518e63" /> - JSON view <img width="2559" alt="Screenshot 2025-01-02 at 13 31 37" src="https://github.com/user-attachments/assets/07575a81-676f-418e-8b98-48eefe11ab00" /> - Rule details page <img width="2555" alt="Screenshot 2025-01-02 at 13 13 16" src="https://github.com/user-attachments/assets/e977b752-9d50-4049-917a-af2e8e3f0dfe" /> <img width="2558" alt="Screenshot 2025-01-02 at 13 14 10" src="https://github.com/user-attachments/assets/06d6f477-5730-48ca-a240-b5e7592bf173" /> ## How to test? - Ensure the `prebuiltRulesCustomizationEnabled` feature flag is enabled - Allow internal APIs via adding `server.restrictInternalApis: false` to `kibana.dev.yaml` - Clear Elasticsearch data - Run Elasticsearch and Kibana locally (do not open Kibana in a web browser) - Install an outdated version of the `security_detection_engine` Fleet package ```bash curl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 2023-10-31" -d '{"force":true}' http://localhost:5601/kbn/api/fleet/epm/packages/security_detection_engine/8.14.1 ``` - Install prebuilt rules ```bash curl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 1" -d '{"mode":"ALL_RULES"}' http://localhost:5601/kbn/internal/detection_engine/prebuilt_rules/installation/_perform ``` - Set "inconsistent" rule schedule for `Suspicious File Creation via Kworker` rule by running a query below ```bash curl -X PATCH --user elastic:changeme -H "Content-Type: application/json" -H "elastic-api-version: 2023-10-31" -H "kbn-xsrf: 123" -d '{"rule_id":"ae343298-97bc-47bc-9ea2-5f2ad831c16e","interval":"10m","from":"now-5m","to":"now-2m"}' http://localhost:5601/kbn/api/detection_engine/rules ``` - Open rule upgrade flyout for `Suspicious File Creation via Kworker` rule --------- Co-authored-by: Elastic Machine <[email protected]>
viduni94
pushed a commit
to viduni94/kibana
that referenced
this pull request
Jan 23, 2025
…lastic#204317) **Fixes: elastic#202715 **Fixes: elastic#204714 ## Summary This PR makes inconsistent/wrong rule's look-back duration prominent for a user. It falls back to a default 1 minute value in rule upgrade workflow. ## Details ### Negative/wrong `lookback` problem There is a difference between rule schedule value in a saved object and value represented to users - Saved object (and rule management API) has `interval`, `from` and `to` fields representing rule schedule. `interval` shows how often a rule runs in task runner. `from` and `to` stored in date math format like `now-10m` represent a date time range used to fetch source events. Task manager strives to run rules exactly every `interval` but it's not always possible due to multiple reasons like system load and various delays. To avoid any gaps to appear `from` point in time usually stands earlier than current time minus `interval`, for example `interval` is `10 minutes` and `from` is `now-12m` meaning rule will analyze events starting from 12 minutes old. `to` represents the latest point in time source events will be analyzed. - Diffable rule and UI represent rule schedule as `interval` and `lookback`. Where `interval` is the same as above and `lookback` and a time duration before current time minus `interval`. For example `interval` is `10 minutes` and lookback is `2 minutes` it means a rule will analyzing events starting with 12 minutes old until the current moment in time. Literally `interval`, `from` and `to` mean a rule runs every `interval` and analyzes events starting from `from` until `to`. Technically `from` and `to` may not have any correlation with `interval`, for example a rule may analyze one year old events. While it's reasonable for manual rule runs and gap remediation the same approach doesn't work well for usual rule schedule. Transformation between `interval`/`from`/`to` and `interval`/`lookback` works only when `to` is equal the current moment in time i.e. `now`. Rule management APIs allow to set any `from` and `to` values resulting in inconsistent rule schedule. Transformed `interval`/`lookback` value won't represent real time interval used to fetch source events for analysis. On top of that negative `lookback` value may puzzle users on the meaning of the negative sign. ### Prebuilt rules with `interval`/`from`/`to` resulting in negative `lookback` Some prebuilt rules have such `interval`, `from` and `to` field values thatnegative `lookback` is expected, for example `Multiple Okta Sessions Detected for a Single User`. It runs every `60 minutes` but has `from` field set to `now-30m` and `to` equals `now`. In the end we have `lookback` equals `to` - `from` - `interval` = `30 minutes` - `60 minutes` = `-30 minutes`. Our UI doesn't handle negative `lookback` values. It simply discards a negative sign and substitutes the rest for editing. In the case above `30 minutes` will be suggested for editing. Saving the form will result in changing `from` to `now-90m` <img width="1712" alt="image" src="https://github.com/user-attachments/assets/05519743-9562-4874-8a73-5596eeccacf2" /> ### Changes in this PR This PR mitigates rule schedule inconsistencies caused by `to` fields not using the current point in time i.e. `now`. The following was done - `DiffableRule`'s `rule_schedule` was changed to have `interval`, `from` and `to` fields instead of `interval` and `lookback` - `_perform` rule upgrade API endpoint was adapted to the new `DIffableRule`'s `rule_schedule` - Rule upgrade flyout calculates and shows `interval` and `lookback` in Diff View, readonly view and field form when `lookback` is non-negative and `to` equals `now` - Rule upgrade flyout shows `interval`, `from` and `to` in Diff View, readonly view and field form when `to` isn't equal `now` or calculated `lookback` is negative - Rule upgrade flyout shows a warning when `to` isn't equal `now` or calculated `lookback` is negative - Rule upgrade flyout's JSON Diff shows `interval` and `lookback` when `lookback` is non-negative and `to` equals `now` and shows `interval`, `from` and `to` in any other case - Rule details page shows `interval`, `from` and `to` in Diff View, readonly view and field form when `to` isn't equal `now` or calculated `lookback` is negative - `maxValue` was added to `ScheduleItemField` to have an ability to restrict input at reasonable values ## Screenshots - Rule upgrade workflow (negative look-back) <img width="2558" alt="Screenshot 2025-01-02 at 13 16 59" src="https://github.com/user-attachments/assets/b8bf727f-11ca-424f-892b-b024ba7f847a" /> <img width="2553" alt="Screenshot 2025-01-02 at 13 17 20" src="https://github.com/user-attachments/assets/9f751ea4-0ce0-4a23-a3b7-0a16494d957e" /> <img width="2558" alt="Screenshot 2025-01-02 at 13 18 24" src="https://github.com/user-attachments/assets/6908ab02-4011-4a6e-85ce-e60d5eac7993" /> - Rule upgrade workflow (positive look-back) <img width="2555" alt="Screenshot 2025-01-02 at 13 19 12" src="https://github.com/user-attachments/assets/06208210-c6cd-4842-8aef-6ade5d13bd36" /> <img width="2558" alt="Screenshot 2025-01-02 at 13 25 31" src="https://github.com/user-attachments/assets/aed38bb0-ccfb-479a-bb3b-e5442c518e63" /> - JSON view <img width="2559" alt="Screenshot 2025-01-02 at 13 31 37" src="https://github.com/user-attachments/assets/07575a81-676f-418e-8b98-48eefe11ab00" /> - Rule details page <img width="2555" alt="Screenshot 2025-01-02 at 13 13 16" src="https://github.com/user-attachments/assets/e977b752-9d50-4049-917a-af2e8e3f0dfe" /> <img width="2558" alt="Screenshot 2025-01-02 at 13 14 10" src="https://github.com/user-attachments/assets/06d6f477-5730-48ca-a240-b5e7592bf173" /> ## How to test? - Ensure the `prebuiltRulesCustomizationEnabled` feature flag is enabled - Allow internal APIs via adding `server.restrictInternalApis: false` to `kibana.dev.yaml` - Clear Elasticsearch data - Run Elasticsearch and Kibana locally (do not open Kibana in a web browser) - Install an outdated version of the `security_detection_engine` Fleet package ```bash curl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 2023-10-31" -d '{"force":true}' http://localhost:5601/kbn/api/fleet/epm/packages/security_detection_engine/8.14.1 ``` - Install prebuilt rules ```bash curl -X POST --user elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 1" -d '{"mode":"ALL_RULES"}' http://localhost:5601/kbn/internal/detection_engine/prebuilt_rules/installation/_perform ``` - Set "inconsistent" rule schedule for `Suspicious File Creation via Kworker` rule by running a query below ```bash curl -X PATCH --user elastic:changeme -H "Content-Type: application/json" -H "elastic-api-version: 2023-10-31" -H "kbn-xsrf: 123" -d '{"rule_id":"ae343298-97bc-47bc-9ea2-5f2ad831c16e","interval":"10m","from":"now-5m","to":"now-2m"}' http://localhost:5601/kbn/api/detection_engine/rules ``` - Open rule upgrade flyout for `Suspicious File Creation via Kworker` rule --------- Co-authored-by: Elastic Machine <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes: #202715
Fixes: #204714
Summary
This PR makes inconsistent/wrong rule's look-back duration prominent for a user. It falls back to a default 1 minute value in rule upgrade workflow.
Details
Negative/wrong
lookbackproblemThere is a difference between rule schedule value in a saved object and value represented to users
interval,fromandtofields representing rule schedule.intervalshows how often a rule runs in task runner.fromandtostored in date math format likenow-10mrepresent a date time range used to fetch source events. Task manager strives to run rules exactly everyintervalbut it's not always possible due to multiple reasons like system load and various delays. To avoid any gaps to appearfrompoint in time usually stands earlier than current time minusinterval, for exampleintervalis10 minutesandfromisnow-12mmeaning rule will analyze events starting from 12 minutes old.torepresents the latest point in time source events will be analyzed.intervalandlookback. Whereintervalis the same as above andlookbackand a time duration before current time minusinterval. For exampleintervalis10 minutesand lookback is2 minutesit means a rule will analyzing events starting with 12 minutes old until the current moment in time.Literally
interval,fromandtomean a rule runs everyintervaland analyzes events starting fromfromuntilto. Technicallyfromandtomay not have any correlation withinterval, for example a rule may analyze one year old events. While it's reasonable for manual rule runs and gap remediation the same approach doesn't work well for usual rule schedule. Transformation betweeninterval/from/toandinterval/lookbackworks only whentois equal the current moment in time i.e.now.Rule management APIs allow to set any
fromandtovalues resulting in inconsistent rule schedule. Transformedinterval/lookbackvalue won't represent real time interval used to fetch source events for analysis. On top of that negativelookbackvalue may puzzle users on the meaning of the negative sign.Prebuilt rules with
interval/from/toresulting in negativelookbackSome prebuilt rules have such
interval,fromandtofield values thatnegativelookbackis expected, for exampleMultiple Okta Sessions Detected for a Single User. It runs every60 minutesbut hasfromfield set tonow-30mandtoequalsnow. In the end we havelookbackequalsto-from-interval=30 minutes-60 minutes=-30 minutes.Our UI doesn't handle negative
lookbackvalues. It simply discards a negative sign and substitutes the rest for editing. In the case above30 minuteswill be suggested for editing. Saving the form will result in changingfromtonow-90mChanges in this PR
This PR mitigates rule schedule inconsistencies caused by
tofields not using the current point in time i.e.now. The following was doneDiffableRule'srule_schedulewas changed to haveinterval,fromandtofields instead ofintervalandlookback_performrule upgrade API endpoint was adapted to the newDIffableRule'srule_scheduleintervalandlookbackin Diff View, readonly view and field form whenlookbackis non-negative andtoequalsnowinterval,fromandtoin Diff View, readonly view and field form whentoisn't equalnowor calculatedlookbackis negativetoisn't equalnowor calculatedlookbackis negativeintervalandlookbackwhenlookbackis non-negative andtoequalsnowand showsinterval,fromandtoin any other caseinterval,fromandtoin Diff View, readonly view and field form whentoisn't equalnowor calculatedlookbackis negativemaxValuewas added toScheduleItemFieldto have an ability to restrict input at reasonable valuesScreenshots
How to test?
prebuiltRulesCustomizationEnabledfeature flag is enabledserver.restrictInternalApis: falsetokibana.dev.yamlsecurity_detection_engineFleet packageSuspicious File Creation via Kworkerrule by running a query belowSuspicious File Creation via Kworkerrule