[Security Solution] Test plan for prebuilt rule customization #204888

dplumlee · 2024-12-19T06:17:13Z

Summary

Addresses #202068

Adds test plan for rule customization features related to the milestone 3 prebuilt rule customization epic

elasticmachine · 2024-12-19T06:17:16Z

Pinging @elastic/security-detections-response (Team:Detections and Resp)

elasticmachine · 2024-12-19T06:17:16Z

Pinging @elastic/security-solution (Team: SecuritySolution)

elasticmachine · 2024-12-19T06:17:17Z

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

banderror · 2024-12-23T19:10:19Z

@pborgonovi Please review this one

maximpn

@dplumlee Thanks for adding Prebuilt Rules Customization test plan 🙏

I reviewed the test plan and left suggestions to make the test plan more robust and improve readability.

It looks like this test plan should include the following scenarios as well

A test scenario (with examples) for each field affecting isCustomized
Custom rules shouldn't have "Modified" badge on rule details page
Custom rules shouldn't have "Modified" badge on rule management table
Scenarios covering ruleSource
Users can navigate to rule edit page from prebuilt rule details

Feel free to reach me to discuss it over Zoom in case of questions.

...ity_solution/docs/testing/test_plans/detection_response/prebuilt_rules/rule_customization.md

maximpn · 2024-12-31T12:06:04Z

...ity_solution/docs/testing/test_plans/detection_response/prebuilt_rules/rule_customization.md

+
+### Terminology
+
+- **Base version**: The version of the rule we ship with the rule package, can be thought of as the "original" version of the rule.


Suggested change

- **Base version**: The version of the rule we ship with the rule package, can be thought of as the "original" version of the rule.

- **Base version**: Prebuilt rule asset we ship in the rule package corresponding to the currently installed prebuilt rules. It represents "original" version of the rule. During prebuilt rules installation prebuilt rule assets data is copied over and becomes an installed prebuilt rule.

Can we say that it's a prebuilt rule asset whose "version" field matches the "version" of an installed rule? WDYT?

...ity_solution/docs/testing/test_plans/detection_response/prebuilt_rules/rule_customization.md

nikitaindik

Thanks for the PR, @dplumlee! Overall, the test coverage looks solid. I’ve left a couple of comments for you to consider.

One potential improvement: should we add a scenario to ensure that the “Edit” buttons in the rule details page and rule management table are clickable when the feature flag is enabled?

Additionally, do we want to include a test to verify that non-modifiable fields, such as version and id are uneditable?”

Looking forward to your thoughts!

…test-plan

dplumlee · 2025-01-10T15:52:17Z

@maximpn @nikitaindik I believe I addressed most of your comments or left questions, take a look when you can.

Scenarios covering ruleSource

@maximpn I think all the additional test cases you mentioned have now been covered but not sure what you meant by this one. I've left a few responses in regard to the ruleSource stuff related to your comments but the whole test plan is kind of related to rule source itself.

nikitaindik

Thanks for addressing my feedback, @dplumlee! The test plan looks good to me now.

pborgonovi

Hey @dplumlee
I've left some comments

...ity_solution/docs/testing/test_plans/detection_response/prebuilt_rules/rule_customization.md

…test-plan

maximpn

@dplumlee Thanks for addressing my comments 👍

I believe ruleSource and is_customized assertions should be removed from test scenarios. Instead of focusing on ruleSource and is_customized test scenarios should focus on Modified badge.

...ity_solution/docs/testing/test_plans/detection_response/prebuilt_rules/rule_customization.md

maximpn · 2025-01-15T10:45:51Z

...ity_solution/docs/testing/test_plans/detection_response/prebuilt_rules/rule_customization.md

+And the rule is unmodified
+When a user edits the rule from the rule edit page to something different than the original version
+Then the rule is successfully updated
+And the ruleSource should be "external"


You may add such ab assumption but it looks like an overkill since the test plan concerns prebuilt rules.

My comment is rather related to unnecessary test scenario explicitness. Users won't see ruleSource in UI. They rather see Modified badge when the rule is modified. The scenario says User can edit a non-customized prebuilt rule from the rule edit page. It literally means user may open rule editing form for a prebuilt rule, change some fields values and save the rule. After that they should see changed values on the rule details page and Modified badge. Checking ruleSource on the way is test implementation details and looks optional.

maximpn · 2025-01-15T10:55:49Z

...ity_solution/docs/testing/test_plans/detection_response/prebuilt_rules/rule_customization.md

+And the action is successfully applied to selected rules
+Then rules that have been changed from their base version should have a ruleSource of "external"
+And isCustomized should be true
+And the "Modified" badge should appear on the respective row in the rule management table


Modified badge is the only visual output here. I think it should stay.

My point is that the test may double check bulk actions affect rule details page as well. Though I gave it a second thought and it looks rather optional.

...ity_solution/docs/testing/test_plans/detection_response/prebuilt_rules/rule_customization.md

maximpn · 2025-01-15T11:01:08Z

...ity_solution/docs/testing/test_plans/detection_response/prebuilt_rules/rule_customization.md

+Then the rule is successfully updated
+And the ruleSource should be "external"
+And isCustomized should be true
+And the "Modified" badge should appear on the rule's detail page


Could we have a separate test scenario to assert Modified badge appears on Rules Update table?

...ity_solution/docs/testing/test_plans/detection_response/prebuilt_rules/rule_customization.md

kibanamachine · 2025-01-16T15:57:38Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12812695257

…c#204888) ## Summary Addresses elastic#202068 Adds test plan for rule customization features related to the milestone 3 prebuilt rule customization epic (cherry picked from commit ded92cf)

kibanamachine · 2025-01-16T16:03:42Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…204888) (#206970) # Backport This will backport the following commits from `main` to `8.x`: - [[Security Solution] Test plan for prebuilt rule customization (#204888)](#204888)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)  Co-authored-by: Davis Plumlee <[email protected]>

…c#204888) ## Summary Addresses elastic#202068 Adds test plan for rule customization features related to the milestone 3 prebuilt rule customization epic

adds test plan

14c226b

dplumlee self-assigned this Dec 19, 2024

dplumlee requested a review from a team as a code owner December 19, 2024 06:17

dplumlee requested a review from maximpn December 19, 2024 06:17

dplumlee added the release_note:skip Skip the PR/issue when compiling release notes label Dec 19, 2024

banderror requested a review from pborgonovi December 23, 2024 19:09

maximpn reviewed Dec 31, 2024

View reviewed changes

nikitaindik self-requested a review January 6, 2025 13:28

nikitaindik reviewed Jan 7, 2025

View reviewed changes

...ity_solution/docs/testing/test_plans/detection_response/prebuilt_rules/rule_customization.md Outdated Show resolved Hide resolved

nikitaindik reviewed Jan 7, 2025

View reviewed changes

dplumlee added 3 commits January 7, 2025 20:44

Merge remote-tracking branch 'upstream/main' into rule-customization-…

c68f262

…test-plan

Merge remote-tracking branch 'upstream/main' into rule-customization-…

8c217b5

…test-plan

addresses comments

bb13567

dplumlee requested review from nikitaindik and maximpn January 10, 2025 15:54

nikitaindik approved these changes Jan 13, 2025

View reviewed changes

pborgonovi reviewed Jan 13, 2025

View reviewed changes

Merge remote-tracking branch 'upstream/main' into rule-customization-…

7418f72

…test-plan

maximpn requested changes Jan 15, 2025

View reviewed changes

dplumlee added 2 commits January 15, 2025 10:13

addresses comments

c6b7994

fixes remaining comments

bb668b5

dplumlee requested review from maximpn and pborgonovi January 15, 2025 15:19

pborgonovi approved these changes Jan 15, 2025

View reviewed changes

maximpn approved these changes Jan 15, 2025

View reviewed changes

dplumlee merged commit ded92cf into elastic:main Jan 16, 2025
8 checks passed

kibanamachine mentioned this pull request Jan 16, 2025

[8.x] [Security Solution] Test plan for prebuilt rule customization (#204888) #206970

Merged

dplumlee deleted the rule-customization-test-plan branch January 16, 2025 16:36

dplumlee mentioned this pull request Jan 17, 2025

[Security Solution] Adds prebuilt rule customization tests #207016

Draft

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution] Test plan for prebuilt rule customization #204888

[Security Solution] Test plan for prebuilt rule customization #204888

dplumlee commented Dec 19, 2024 •

edited by kibanamachine

Loading

elasticmachine commented Dec 19, 2024

elasticmachine commented Dec 19, 2024

elasticmachine commented Dec 19, 2024

banderror commented Dec 23, 2024

maximpn left a comment •

edited

Loading

maximpn Dec 31, 2024

nikitaindik Jan 7, 2025

nikitaindik left a comment

dplumlee commented Jan 10, 2025

nikitaindik left a comment

pborgonovi left a comment

maximpn left a comment

maximpn Jan 15, 2025

maximpn Jan 15, 2025

maximpn Jan 15, 2025

kibanamachine commented Jan 16, 2025

kibanamachine commented Jan 16, 2025


		### Terminology

		- Base version: The version of the rule we ship with the rule package, can be thought of as the "original" version of the rule.

	- Base version: The version of the rule we ship with the rule package, can be thought of as the "original" version of the rule.
	- Base version: Prebuilt rule asset we ship in the rule package corresponding to the currently installed prebuilt rules. It represents "original" version of the rule. During prebuilt rules installation prebuilt rule assets data is copied over and becomes an installed prebuilt rule.

[Security Solution] Test plan for prebuilt rule customization #204888

[Security Solution] Test plan for prebuilt rule customization #204888

Conversation

dplumlee commented Dec 19, 2024 • edited by kibanamachine Loading

Summary

elasticmachine commented Dec 19, 2024

elasticmachine commented Dec 19, 2024

elasticmachine commented Dec 19, 2024

banderror commented Dec 23, 2024

maximpn left a comment • edited Loading

Choose a reason for hiding this comment

maximpn Dec 31, 2024

Choose a reason for hiding this comment

nikitaindik Jan 7, 2025

Choose a reason for hiding this comment

nikitaindik left a comment

Choose a reason for hiding this comment

dplumlee commented Jan 10, 2025

nikitaindik left a comment

Choose a reason for hiding this comment

pborgonovi left a comment

Choose a reason for hiding this comment

maximpn left a comment

Choose a reason for hiding this comment

maximpn Jan 15, 2025

Choose a reason for hiding this comment

maximpn Jan 15, 2025

Choose a reason for hiding this comment

maximpn Jan 15, 2025

Choose a reason for hiding this comment

kibanamachine commented Jan 16, 2025

kibanamachine commented Jan 16, 2025

💚 All backports created successfully

Questions ?

dplumlee commented Dec 19, 2024 •

edited by kibanamachine

Loading

maximpn left a comment •

edited

Loading