Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution] Fix inability to unset optional field values #204231

Merged

Conversation

maximpn
Copy link
Contributor

@maximpn maximpn commented Dec 13, 2024

Resolves: #203634

Summary

This PR fixes bugs blocking unsetting optional rule field values in rule upgrade workflow.

Details

Changes here cover 3 groups of fields optional, string fields allowing empty strings and array fields allowing empty arrays. It was verified that fields in that groups allow to unset the value.

The following issues were fixed

  • inability to set an empty string or setup and note fields
    It required adding stripEmptyFields: false for rule upgrade fields edit form.
  • inability to unset timestamp_override field
    Timestamp override form deserializer was fixed.
  • inability to unset alert_suppression
    Alert Suppression was excluded from special special fields list always upgrading to the current value. It's expected Alert Suppression won't be included in Prebuilt Rules delivered in prebuilt rules packages. The only way to get this setting and have it included in rule upgrade flyout is editing a prebuilt rule by a user with a sufficient licence.

The following fields were verified and fixed where necessary

Optional fields

  • investigation_fields
  • rule_name_override
  • ⚠️ timestamp_override (field's form deserializer was fixed)
  • timeline_template
  • building_block
  • ⚠️ alert_suppression (the field was excluded from special special fields list always upgrading to the current value)
  • threat_indicator_path (empty value resets to default threat.indicator)

String fields allowing empty strings

  • ⚠️ note (required adding stripEmptyFields: false to the form)
  • ⚠️ setup (required adding stripEmptyFields: false to the form)

Array fields allowing empty arrays

  • tags
  • references
  • false_positives
  • threat
  • related_integrations
  • required_fields
  • severity_mapping
  • risk_score_mapping

Screenshots

Screenshot 2024-12-17 at 09 15 14

Screen.Recording.2024-12-17.at.09.10.29.mov

How to test?

  • Ensure the prebuiltRulesCustomizationEnabled feature flag is enabled
  • Allow internal APIs via adding server.restrictInternalApis: false to kibana.dev.yaml
  • Clear Elasticsearch data
  • Run Elasticsearch and Kibana locally (do not open Kibana in a web browser)
  • Install an outdated version of the security_detection_engine Fleet package
curl -X POST --user elastic:changeme  -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 2023-10-31" -d '{"force":true}' http://localhost:5601/kbn/api/fleet/epm/packages/security_detection_engine/8.14.1
  • Install prebuilt rules
curl -X POST --user elastic:changeme  -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 1" -d '{"mode":"ALL_RULES"}' http://localhost:5601/kbn/internal/detection_engine/prebuilt_rules/installation/_perform
  • Customize one or more rules (change fields to see them in rule upgrade workflow)
  • Open Rule upgrade for the rule(s)
  • Unset field values
  • Upgrade rule(s)

@maximpn maximpn added release_note:skip Skip the PR/issue when compiling release notes impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. v9.0.0 Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Team:Detection Rule Management Security Detection Rule Management Team Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area backport:version Backport to applied version labels v8.18.0 labels Dec 13, 2024
@maximpn maximpn self-assigned this Dec 13, 2024
@maximpn maximpn requested a review from banderror December 13, 2024 16:01
@maximpn maximpn marked this pull request as ready for review December 13, 2024 16:40
@maximpn maximpn requested a review from a team as a code owner December 13, 2024 16:40
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

@banderror
Copy link
Contributor

@maximpn Please attach screenshots or video that would demonstrate the fix.

@maximpn maximpn force-pushed the fix-inability-to-unset-optional-field-values branch from 12c9bda to ea12f5e Compare December 17, 2024 07:56
@maximpn
Copy link
Contributor Author

maximpn commented Dec 17, 2024

@banderror I attached a screenshot and a video to the description.

@banderror
Copy link
Contributor

@elasticmachine merge upstream

Copy link
Contributor

@banderror banderror left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Although I'm not familiar with the ThreeWayDiff components area in the code, changes look very simple and good 👍

Tested the following fields locally by customizing them in prebuilt rules and then resetting to empty values on upgrade. Everything worked great:

  • Optional fields
    • 🟢 investigation_fields
    • 🟢 rule_name_override
    • 🟢 timestamp_override
    • 🟢 timeline_template
    • 🟢 building_block
    • 🟢 alert_suppression
  • String fields allowing empty strings
    • 🟢 note
    • 🟢 setup
  • Array fields allowing empty arrays
    • 🟢 tags
    • 🟢 references
    • 🟢 false_positives
    • 🟢 threat
    • 🟢 related_integrations
    • 🟢 required_fields
    • 🟢 severity_mapping
    • 🟢 risk_score_mapping

I didn't test:

  • machine_learning_job_id
  • threat_indicator_path
  • any other type-specific fields

Thank you @maximpn for catching and fixing this issue 🚀

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

Metrics [docs]

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id before after diff
securitySolution 21.4MB 21.4MB +90.0B

History

cc @maximpn

@banderror banderror merged commit 54989a5 into elastic:main Dec 20, 2024
8 checks passed
@kibanamachine
Copy link
Contributor

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12431406203

kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Dec 20, 2024
…stic#204231)

**Resolves: elastic#203634

## Summary

This PR fixes bugs blocking unsetting optional rule field values in rule
upgrade workflow.

## Details

Changes here cover 3 groups of fields optional, string fields allowing
empty strings and array fields allowing empty arrays. It was verified
that fields in that groups allow to unset the value.

The following issues were fixed
- inability to set an empty string or `setup` and `note` fields
It required adding `stripEmptyFields: false` for rule upgrade fields
edit form.
- inability to unset `timestamp_override` field
  Timestamp override form deserializer was fixed.
- inability to unset `alert_suppression`
Alert Suppression was excluded from special special fields list always
upgrading to the current value. It's expected Alert Suppression won't be
included in Prebuilt Rules delivered in prebuilt rules packages. The
only way to get this setting and have it included in rule upgrade flyout
is editing a prebuilt rule by a user with a sufficient licence.

The following fields were verified and fixed where necessary

### Optional fields

- ✅ `investigation_fields`
- ✅ `rule_name_override`
- ⚠️ `timestamp_override` (field's form deserializer was fixed)
- ✅ `timeline_template`
- ✅ `building_block`
- ⚠️ `alert_suppression` (the field was excluded from special special
fields list always upgrading to the current value)
- ✅ `threat_indicator_path` (empty value resets to default
`threat.indicator`)

### String fields allowing empty strings

- ⚠️ `note` (required adding `stripEmptyFields: false` to the form)
- ⚠️ `setup` (required adding `stripEmptyFields: false` to the form)

### Array fields allowing empty arrays

- ✅ `tags`
- ✅ `references`
- ✅ `false_positives`
- ✅ `threat`
- ✅ `related_integrations`
- ✅ `required_fields`
- ✅ `severity_mapping`
- ✅ `risk_score_mapping`

## Screenshots

![Screenshot 2024-12-17 at 09 15
14](https://github.com/user-attachments/assets/671f5198-55da-4899-ab52-1e93f3c841af)

https://github.com/user-attachments/assets/bd36e5ba-e7fb-4733-a792-ea5435d579e2

## How to test?

- Ensure the `prebuiltRulesCustomizationEnabled` feature flag is enabled
- Allow internal APIs via adding `server.restrictInternalApis: false` to
`kibana.dev.yaml`
- Clear Elasticsearch data
- Run Elasticsearch and Kibana locally (do not open Kibana in a web
browser)
- Install an outdated version of the `security_detection_engine` Fleet
package
```bash
curl -X POST --user elastic:changeme  -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 2023-10-31" -d '{"force":true}' http://localhost:5601/kbn/api/fleet/epm/packages/security_detection_engine/8.14.1
```

- Install prebuilt rules
```bash
curl -X POST --user elastic:changeme  -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 1" -d '{"mode":"ALL_RULES"}' http://localhost:5601/kbn/internal/detection_engine/prebuilt_rules/installation/_perform
```

- Customize one or more rules (change fields to see them in rule upgrade
workflow)
- Open Rule upgrade for the rule(s)
- Unset field values
- Upgrade rule(s)

---------

Co-authored-by: Elastic Machine <[email protected]>
(cherry picked from commit 54989a5)
@kibanamachine
Copy link
Contributor

💚 All backports created successfully

Status Branch Result
8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

kibanamachine added a commit that referenced this pull request Dec 20, 2024
#204231) (#205041)

# Backport

This will backport the following commits from `main` to `8.x`:
- [[Security Solution] Fix inability to unset optional field values
(#204231)](#204231)

<!--- Backport version: 9.4.3 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Maxim
Palenov","email":"[email protected]"},"sourceCommit":{"committedDate":"2024-12-20T12:26:50Z","message":"[Security
Solution] Fix inability to unset optional field values
(#204231)\n\n**Resolves:
https://github.com/elastic/kibana/issues/203634**\r\n\r\n##
Summary\r\n\r\nThis PR fixes bugs blocking unsetting optional rule field
values in rule\r\nupgrade workflow.\r\n\r\n## Details\r\n\r\nChanges
here cover 3 groups of fields optional, string fields allowing\r\nempty
strings and array fields allowing empty arrays. It was verified\r\nthat
fields in that groups allow to unset the value.\r\n\r\nThe following
issues were fixed\r\n- inability to set an empty string or `setup` and
`note` fields\r\nIt required adding `stripEmptyFields: false` for rule
upgrade fields\r\nedit form.\r\n- inability to unset
`timestamp_override` field\r\n Timestamp override form deserializer was
fixed.\r\n- inability to unset `alert_suppression`\r\nAlert Suppression
was excluded from special special fields list always\r\nupgrading to the
current value. It's expected Alert Suppression won't be\r\nincluded in
Prebuilt Rules delivered in prebuilt rules packages. The\r\nonly way to
get this setting and have it included in rule upgrade flyout\r\nis
editing a prebuilt rule by a user with a sufficient licence.\r\n\r\nThe
following fields were verified and fixed where necessary\r\n\r\n###
Optional fields\r\n\r\n- ✅ `investigation_fields`\r\n- ✅
`rule_name_override`\r\n- ⚠️ `timestamp_override` (field's form
deserializer was fixed)\r\n- ✅ `timeline_template`\r\n- ✅
`building_block`\r\n- ⚠️ `alert_suppression` (the field was excluded
from special special\r\nfields list always upgrading to the current
value)\r\n- ✅ `threat_indicator_path` (empty value resets to
default\r\n`threat.indicator`)\r\n\r\n### String fields allowing empty
strings\r\n\r\n- ⚠️ `note` (required adding `stripEmptyFields: false` to
the form)\r\n- ⚠️ `setup` (required adding `stripEmptyFields: false` to
the form)\r\n\r\n### Array fields allowing empty arrays\r\n\r\n- ✅
`tags`\r\n- ✅ `references`\r\n- ✅ `false_positives`\r\n- ✅ `threat`\r\n-
✅ `related_integrations`\r\n- ✅ `required_fields`\r\n- ✅
`severity_mapping`\r\n- ✅ `risk_score_mapping`\r\n\r\n##
Screenshots\r\n\r\n![Screenshot 2024-12-17 at 09
15\r\n14](https://github.com/user-attachments/assets/671f5198-55da-4899-ab52-1e93f3c841af)\r\n\r\n\r\nhttps://github.com/user-attachments/assets/bd36e5ba-e7fb-4733-a792-ea5435d579e2\r\n\r\n##
How to test?\r\n\r\n- Ensure the `prebuiltRulesCustomizationEnabled`
feature flag is enabled\r\n- Allow internal APIs via adding
`server.restrictInternalApis: false` to\r\n`kibana.dev.yaml`\r\n- Clear
Elasticsearch data\r\n- Run Elasticsearch and Kibana locally (do not
open Kibana in a web\r\nbrowser)\r\n- Install an outdated version of the
`security_detection_engine` Fleet\r\npackage\r\n```bash\r\ncurl -X POST
--user elastic:changeme -H 'Content-Type: application/json' -H
'kbn-xsrf: 123' -H \"elastic-api-version: 2023-10-31\" -d
'{\"force\":true}'
http://localhost:5601/kbn/api/fleet/epm/packages/security_detection_engine/8.14.1\r\n```\r\n\r\n-
Install prebuilt rules\r\n```bash\r\ncurl -X POST --user
elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123'
-H \"elastic-api-version: 1\" -d '{\"mode\":\"ALL_RULES\"}'
http://localhost:5601/kbn/internal/detection_engine/prebuilt_rules/installation/_perform\r\n```\r\n\r\n-
Customize one or more rules (change fields to see them in rule
upgrade\r\nworkflow)\r\n- Open Rule upgrade for the rule(s)\r\n- Unset
field values\r\n- Upgrade
rule(s)\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine
<[email protected]>","sha":"54989a519260397f26694be0db1913a7468b40cb","branchLabelMapping":{"^v9.0.0$":"main","^v8.18.0$":"8.x","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","impact:high","v9.0.0","Team:Detections
and Resp","Team: SecuritySolution","Team:Detection Rule
Management","Feature:Prebuilt Detection
Rules","backport:version","v8.18.0"],"title":"[Security Solution] Fix
inability to unset optional field
values","number":204231,"url":"https://github.com/elastic/kibana/pull/204231","mergeCommit":{"message":"[Security
Solution] Fix inability to unset optional field values
(#204231)\n\n**Resolves:
https://github.com/elastic/kibana/issues/203634**\r\n\r\n##
Summary\r\n\r\nThis PR fixes bugs blocking unsetting optional rule field
values in rule\r\nupgrade workflow.\r\n\r\n## Details\r\n\r\nChanges
here cover 3 groups of fields optional, string fields allowing\r\nempty
strings and array fields allowing empty arrays. It was verified\r\nthat
fields in that groups allow to unset the value.\r\n\r\nThe following
issues were fixed\r\n- inability to set an empty string or `setup` and
`note` fields\r\nIt required adding `stripEmptyFields: false` for rule
upgrade fields\r\nedit form.\r\n- inability to unset
`timestamp_override` field\r\n Timestamp override form deserializer was
fixed.\r\n- inability to unset `alert_suppression`\r\nAlert Suppression
was excluded from special special fields list always\r\nupgrading to the
current value. It's expected Alert Suppression won't be\r\nincluded in
Prebuilt Rules delivered in prebuilt rules packages. The\r\nonly way to
get this setting and have it included in rule upgrade flyout\r\nis
editing a prebuilt rule by a user with a sufficient licence.\r\n\r\nThe
following fields were verified and fixed where necessary\r\n\r\n###
Optional fields\r\n\r\n- ✅ `investigation_fields`\r\n- ✅
`rule_name_override`\r\n- ⚠️ `timestamp_override` (field's form
deserializer was fixed)\r\n- ✅ `timeline_template`\r\n- ✅
`building_block`\r\n- ⚠️ `alert_suppression` (the field was excluded
from special special\r\nfields list always upgrading to the current
value)\r\n- ✅ `threat_indicator_path` (empty value resets to
default\r\n`threat.indicator`)\r\n\r\n### String fields allowing empty
strings\r\n\r\n- ⚠️ `note` (required adding `stripEmptyFields: false` to
the form)\r\n- ⚠️ `setup` (required adding `stripEmptyFields: false` to
the form)\r\n\r\n### Array fields allowing empty arrays\r\n\r\n- ✅
`tags`\r\n- ✅ `references`\r\n- ✅ `false_positives`\r\n- ✅ `threat`\r\n-
✅ `related_integrations`\r\n- ✅ `required_fields`\r\n- ✅
`severity_mapping`\r\n- ✅ `risk_score_mapping`\r\n\r\n##
Screenshots\r\n\r\n![Screenshot 2024-12-17 at 09
15\r\n14](https://github.com/user-attachments/assets/671f5198-55da-4899-ab52-1e93f3c841af)\r\n\r\n\r\nhttps://github.com/user-attachments/assets/bd36e5ba-e7fb-4733-a792-ea5435d579e2\r\n\r\n##
How to test?\r\n\r\n- Ensure the `prebuiltRulesCustomizationEnabled`
feature flag is enabled\r\n- Allow internal APIs via adding
`server.restrictInternalApis: false` to\r\n`kibana.dev.yaml`\r\n- Clear
Elasticsearch data\r\n- Run Elasticsearch and Kibana locally (do not
open Kibana in a web\r\nbrowser)\r\n- Install an outdated version of the
`security_detection_engine` Fleet\r\npackage\r\n```bash\r\ncurl -X POST
--user elastic:changeme -H 'Content-Type: application/json' -H
'kbn-xsrf: 123' -H \"elastic-api-version: 2023-10-31\" -d
'{\"force\":true}'
http://localhost:5601/kbn/api/fleet/epm/packages/security_detection_engine/8.14.1\r\n```\r\n\r\n-
Install prebuilt rules\r\n```bash\r\ncurl -X POST --user
elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123'
-H \"elastic-api-version: 1\" -d '{\"mode\":\"ALL_RULES\"}'
http://localhost:5601/kbn/internal/detection_engine/prebuilt_rules/installation/_perform\r\n```\r\n\r\n-
Customize one or more rules (change fields to see them in rule
upgrade\r\nworkflow)\r\n- Open Rule upgrade for the rule(s)\r\n- Unset
field values\r\n- Upgrade
rule(s)\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine
<[email protected]>","sha":"54989a519260397f26694be0db1913a7468b40cb"}},"sourceBranch":"main","suggestedTargetBranches":["8.x"],"targetPullRequestStates":[{"branch":"main","label":"v9.0.0","branchLabelMappingKey":"^v9.0.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/204231","number":204231,"mergeCommit":{"message":"[Security
Solution] Fix inability to unset optional field values
(#204231)\n\n**Resolves:
https://github.com/elastic/kibana/issues/203634**\r\n\r\n##
Summary\r\n\r\nThis PR fixes bugs blocking unsetting optional rule field
values in rule\r\nupgrade workflow.\r\n\r\n## Details\r\n\r\nChanges
here cover 3 groups of fields optional, string fields allowing\r\nempty
strings and array fields allowing empty arrays. It was verified\r\nthat
fields in that groups allow to unset the value.\r\n\r\nThe following
issues were fixed\r\n- inability to set an empty string or `setup` and
`note` fields\r\nIt required adding `stripEmptyFields: false` for rule
upgrade fields\r\nedit form.\r\n- inability to unset
`timestamp_override` field\r\n Timestamp override form deserializer was
fixed.\r\n- inability to unset `alert_suppression`\r\nAlert Suppression
was excluded from special special fields list always\r\nupgrading to the
current value. It's expected Alert Suppression won't be\r\nincluded in
Prebuilt Rules delivered in prebuilt rules packages. The\r\nonly way to
get this setting and have it included in rule upgrade flyout\r\nis
editing a prebuilt rule by a user with a sufficient licence.\r\n\r\nThe
following fields were verified and fixed where necessary\r\n\r\n###
Optional fields\r\n\r\n- ✅ `investigation_fields`\r\n- ✅
`rule_name_override`\r\n- ⚠️ `timestamp_override` (field's form
deserializer was fixed)\r\n- ✅ `timeline_template`\r\n- ✅
`building_block`\r\n- ⚠️ `alert_suppression` (the field was excluded
from special special\r\nfields list always upgrading to the current
value)\r\n- ✅ `threat_indicator_path` (empty value resets to
default\r\n`threat.indicator`)\r\n\r\n### String fields allowing empty
strings\r\n\r\n- ⚠️ `note` (required adding `stripEmptyFields: false` to
the form)\r\n- ⚠️ `setup` (required adding `stripEmptyFields: false` to
the form)\r\n\r\n### Array fields allowing empty arrays\r\n\r\n- ✅
`tags`\r\n- ✅ `references`\r\n- ✅ `false_positives`\r\n- ✅ `threat`\r\n-
✅ `related_integrations`\r\n- ✅ `required_fields`\r\n- ✅
`severity_mapping`\r\n- ✅ `risk_score_mapping`\r\n\r\n##
Screenshots\r\n\r\n![Screenshot 2024-12-17 at 09
15\r\n14](https://github.com/user-attachments/assets/671f5198-55da-4899-ab52-1e93f3c841af)\r\n\r\n\r\nhttps://github.com/user-attachments/assets/bd36e5ba-e7fb-4733-a792-ea5435d579e2\r\n\r\n##
How to test?\r\n\r\n- Ensure the `prebuiltRulesCustomizationEnabled`
feature flag is enabled\r\n- Allow internal APIs via adding
`server.restrictInternalApis: false` to\r\n`kibana.dev.yaml`\r\n- Clear
Elasticsearch data\r\n- Run Elasticsearch and Kibana locally (do not
open Kibana in a web\r\nbrowser)\r\n- Install an outdated version of the
`security_detection_engine` Fleet\r\npackage\r\n```bash\r\ncurl -X POST
--user elastic:changeme -H 'Content-Type: application/json' -H
'kbn-xsrf: 123' -H \"elastic-api-version: 2023-10-31\" -d
'{\"force\":true}'
http://localhost:5601/kbn/api/fleet/epm/packages/security_detection_engine/8.14.1\r\n```\r\n\r\n-
Install prebuilt rules\r\n```bash\r\ncurl -X POST --user
elastic:changeme -H 'Content-Type: application/json' -H 'kbn-xsrf: 123'
-H \"elastic-api-version: 1\" -d '{\"mode\":\"ALL_RULES\"}'
http://localhost:5601/kbn/internal/detection_engine/prebuilt_rules/installation/_perform\r\n```\r\n\r\n-
Customize one or more rules (change fields to see them in rule
upgrade\r\nworkflow)\r\n- Open Rule upgrade for the rule(s)\r\n- Unset
field values\r\n- Upgrade
rule(s)\r\n\r\n---------\r\n\r\nCo-authored-by: Elastic Machine
<[email protected]>","sha":"54989a519260397f26694be0db1913a7468b40cb"}},{"branch":"8.x","label":"v8.18.0","branchLabelMappingKey":"^v8.18.0$","isSourceBranch":false,"state":"NOT_CREATED"}]}]
BACKPORT-->

Co-authored-by: Maxim Palenov <[email protected]>
@maximpn maximpn deleted the fix-inability-to-unset-optional-field-values branch December 25, 2024 22:25
stratoula pushed a commit to stratoula/kibana that referenced this pull request Jan 2, 2025
…stic#204231)

**Resolves: elastic#203634

## Summary

This PR fixes bugs blocking unsetting optional rule field values in rule
upgrade workflow.

## Details

Changes here cover 3 groups of fields optional, string fields allowing
empty strings and array fields allowing empty arrays. It was verified
that fields in that groups allow to unset the value.

The following issues were fixed
- inability to set an empty string or `setup` and `note` fields
It required adding `stripEmptyFields: false` for rule upgrade fields
edit form.
- inability to unset `timestamp_override` field
  Timestamp override form deserializer was fixed.
- inability to unset `alert_suppression`
Alert Suppression was excluded from special special fields list always
upgrading to the current value. It's expected Alert Suppression won't be
included in Prebuilt Rules delivered in prebuilt rules packages. The
only way to get this setting and have it included in rule upgrade flyout
is editing a prebuilt rule by a user with a sufficient licence.

The following fields were verified and fixed where necessary

### Optional fields

- ✅ `investigation_fields`
- ✅ `rule_name_override`
- ⚠️ `timestamp_override` (field's form deserializer was fixed)
- ✅ `timeline_template`
- ✅ `building_block`
- ⚠️ `alert_suppression` (the field was excluded from special special
fields list always upgrading to the current value)
- ✅ `threat_indicator_path` (empty value resets to default
`threat.indicator`)

### String fields allowing empty strings

- ⚠️ `note` (required adding `stripEmptyFields: false` to the form)
- ⚠️ `setup` (required adding `stripEmptyFields: false` to the form)

### Array fields allowing empty arrays

- ✅ `tags`
- ✅ `references`
- ✅ `false_positives`
- ✅ `threat`
- ✅ `related_integrations`
- ✅ `required_fields`
- ✅ `severity_mapping`
- ✅ `risk_score_mapping`

## Screenshots

![Screenshot 2024-12-17 at 09 15
14](https://github.com/user-attachments/assets/671f5198-55da-4899-ab52-1e93f3c841af)


https://github.com/user-attachments/assets/bd36e5ba-e7fb-4733-a792-ea5435d579e2

## How to test?

- Ensure the `prebuiltRulesCustomizationEnabled` feature flag is enabled
- Allow internal APIs via adding `server.restrictInternalApis: false` to
`kibana.dev.yaml`
- Clear Elasticsearch data
- Run Elasticsearch and Kibana locally (do not open Kibana in a web
browser)
- Install an outdated version of the `security_detection_engine` Fleet
package
```bash
curl -X POST --user elastic:changeme  -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 2023-10-31" -d '{"force":true}' http://localhost:5601/kbn/api/fleet/epm/packages/security_detection_engine/8.14.1
```

- Install prebuilt rules
```bash
curl -X POST --user elastic:changeme  -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 1" -d '{"mode":"ALL_RULES"}' http://localhost:5601/kbn/internal/detection_engine/prebuilt_rules/installation/_perform
```

- Customize one or more rules (change fields to see them in rule upgrade
workflow)
- Open Rule upgrade for the rule(s)
- Unset field values
- Upgrade rule(s)

---------

Co-authored-by: Elastic Machine <[email protected]>
benakansara pushed a commit to benakansara/kibana that referenced this pull request Jan 2, 2025
…stic#204231)

**Resolves: elastic#203634

## Summary

This PR fixes bugs blocking unsetting optional rule field values in rule
upgrade workflow.

## Details

Changes here cover 3 groups of fields optional, string fields allowing
empty strings and array fields allowing empty arrays. It was verified
that fields in that groups allow to unset the value.

The following issues were fixed
- inability to set an empty string or `setup` and `note` fields
It required adding `stripEmptyFields: false` for rule upgrade fields
edit form.
- inability to unset `timestamp_override` field
  Timestamp override form deserializer was fixed.
- inability to unset `alert_suppression`
Alert Suppression was excluded from special special fields list always
upgrading to the current value. It's expected Alert Suppression won't be
included in Prebuilt Rules delivered in prebuilt rules packages. The
only way to get this setting and have it included in rule upgrade flyout
is editing a prebuilt rule by a user with a sufficient licence.

The following fields were verified and fixed where necessary

### Optional fields

- ✅ `investigation_fields`
- ✅ `rule_name_override`
- ⚠️ `timestamp_override` (field's form deserializer was fixed)
- ✅ `timeline_template`
- ✅ `building_block`
- ⚠️ `alert_suppression` (the field was excluded from special special
fields list always upgrading to the current value)
- ✅ `threat_indicator_path` (empty value resets to default
`threat.indicator`)

### String fields allowing empty strings

- ⚠️ `note` (required adding `stripEmptyFields: false` to the form)
- ⚠️ `setup` (required adding `stripEmptyFields: false` to the form)

### Array fields allowing empty arrays

- ✅ `tags`
- ✅ `references`
- ✅ `false_positives`
- ✅ `threat`
- ✅ `related_integrations`
- ✅ `required_fields`
- ✅ `severity_mapping`
- ✅ `risk_score_mapping`

## Screenshots

![Screenshot 2024-12-17 at 09 15
14](https://github.com/user-attachments/assets/671f5198-55da-4899-ab52-1e93f3c841af)


https://github.com/user-attachments/assets/bd36e5ba-e7fb-4733-a792-ea5435d579e2

## How to test?

- Ensure the `prebuiltRulesCustomizationEnabled` feature flag is enabled
- Allow internal APIs via adding `server.restrictInternalApis: false` to
`kibana.dev.yaml`
- Clear Elasticsearch data
- Run Elasticsearch and Kibana locally (do not open Kibana in a web
browser)
- Install an outdated version of the `security_detection_engine` Fleet
package
```bash
curl -X POST --user elastic:changeme  -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 2023-10-31" -d '{"force":true}' http://localhost:5601/kbn/api/fleet/epm/packages/security_detection_engine/8.14.1
```

- Install prebuilt rules
```bash
curl -X POST --user elastic:changeme  -H 'Content-Type: application/json' -H 'kbn-xsrf: 123' -H "elastic-api-version: 1" -d '{"mode":"ALL_RULES"}' http://localhost:5601/kbn/internal/detection_engine/prebuilt_rules/installation/_perform
```

- Customize one or more rules (change fields to see them in rule upgrade
workflow)
- Open Rule upgrade for the rule(s)
- Unset field values
- Upgrade rule(s)

---------

Co-authored-by: Elastic Machine <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport:version Backport to applied version labels Feature:Prebuilt Detection Rules Security Solution Prebuilt Detection Rules area impact:high Addressing this issue will have a high level of impact on the quality/strength of our product. release_note:skip Skip the PR/issue when compiling release notes Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.18.0 v9.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Security Solution] Unable to unset field value upon rules upgrade
4 participants