[Security Solution] Fields Displayed in Current Version Despite Being Removed

[pborgonovi]

Describe the bug:

When fields like Tags, Related Integrations, and MITRE ATT&CK are removed in the Customized version of a rule, they still appear in the Current version within the rule updates table.

Kibana/Elasticsearch Stack version:

8.x

Current branch: 8.x  
Latest commit: d0c9a2f1f52 - [8.x] [Stack Monitoring / Logs] Fix Stack Monitoring logs links (#200043) (#200227)  
Remote tracking: origin/8.x  
Status relative to remote: up to date (no pending commits)  

Server OS version:

Browser and Browser OS versions:

Elastic Endpoint version:

Original install method (e.g. download page, yum, from source, etc.):

Functional Area (e.g. Endpoint management, timelines, resolver, etc.):

Steps to reproduce:

1. Select a prebuilt rule which has an update available and customize it by removing all tags in the Tags, Related Integrations and MITRE ATT&CK fields.
2. Save the customized rule.
3. Open the Rule Updates table for the customized rule.
4. Observe the tags displayed under the Current version.

Current behavior:

Fields that were cleared (e.g., Tags, Related Integrations, MITRE ATT&CK) in the Customized version still appear with their previous values in the Current version within the rule updates table.

Expected behavior:

The Current version should correctly reflect the actual customized state of the rule. For fields like Tags, Related Integrations, or MITRE ATT&CK, if all values are removed during customization, the Current version should display an empty state or indicate that the field is cleared.

Screenshots (if relevant):

Screen.Recording.2024-11-14.at.3.57.45.PM.mov

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context (logs, chat logs, magical formulas, etc.):

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[banderror]

@pborgonovi Here we have two things to align on.

First, just like with #200071, there is misunderstanding of how these diffs work. In your video from the description, the Current vs Final view shows that in the Final version these tags are added to an empty array of tags, not as you say "still appear with their previous values in the Current version within the rule updates table". The diff visualization is correct.

Second, the actual question is why these tags are present in the Final version in the first place? The situation is the following:

• In the Base version you had some tags, e.g. ['foo', 'bar']
• In the Current version you have an empty array of tags [], because you customized the rule by removing all tags from it.
• In the Target version from Elastic, probably these tags were not changed: ['foo', 'bar']
• This means that in the Final version suggested by the upgrade logic it should have been [] instead of ['foo', 'bar']
• And so the Current vs Final diff on the left side should have been empty.

I'm sure that this didn't happen because the Base version didn't exist. Without the Base version the diff algorithm couldn't identify that it needs to keep your customization in the Final version, and picked the Target version instead.

The reason for the missing Base version is: our Fleet package with prebuilt rules currently doesn't ship all historical versions of prebuilt rules. We're working with the TRADE team on fixing this (#187645, elastic/detection-rules#4150 (comment)) and this is a release blocker. You can read more about this issue in this slack thread (internal).

Let's close this ticket. I will open a more specific bug about missing Base versions.

[banderror]

Replacement bug: #201500