[Security Solution] Inform users about missing base versions during prebuilt rule upgrade

[banderror]

Epics: https://github.com/elastic/security-team/issues/1974, #179907
Context: #184889 (comment)

Summary

When user upgrades a prebuilt rule that can be upgraded to a new version, and the base version for this rule is missing, we should consider warning the user about that and explaining to the user how it will affect the upgrade workflow.

Background

For every rule that can be upgraded we rely on 3 versions of it:

• The current version is the currently installed prebuilt rule. Corresponds to its alerting rule saved object. Can be customized by the user, or can be left as is after installation.
• The target version is the version we're going to upgrade to - the lates available rule version from Elastic. Corresponds to a security-rule saved object of the target version.
• The base version is the security-rule saved object that has the same rule.version as the currently installed rule. In this object we can find the original, non-customized rule attributes.

In order to have security-rule saved objects in a cluster, the package with prebuilt rule assets needs to be installed. In certain situations these assets can be missing, and we need to handle this in our logic.

There can be a few ways to loose rule assets partially which can lead to having a target version, but missing the base one:

• By manually deleting some of them using a script, or dev tools, etc. I think chances are very low, and that would be the shooting yourself in the foot type of case.
• By installing a new package version where some of the historical rule versions will be missing. We have a hard limit of 15000 rule versions per package. We haven't reached it yet, but reaching it is a matter of time. When the TRaDE team reaches it, they will need to come up with an algorithm for evicting old rule versions from the package. Our upgrade logic should work well regardless of existence or non-existence of certain historical versions. What are the chances for the user to find themselves in this situation? I think it will depend on this eviction algorithm in the future, but currently I think it's 0.

From the UX perspective, we have a trade-off that we could handle in a few ways:

• 1st option is to treat -AA as ABB and -AB as ABC. The trade-off is safety at the cost of usability. In practice, all -AA and -AB fields would be shown in the rule upgrade flyout, with -AB fields shown as conflicting. This might look confusing, dangerous and overwhelming, especially if the user has never customized their installed rules and just lost some of the prebuilt rule assets.
• 2nd option is to treat -AA as AAA and -AB as AAB. The trade-off is usability at the cost of safety. In practice, all -AA fields would not be shown in the rule upgrade flyout, and -AB fields would be shown in the flyout as auto-accepted, thus significantly saving the time to upgrade the rules. This might be dangerous for those rules that had been customized by the user, then Elastic published a new package with updates to those rules, and the user lost some of the prebuilt rule assets (-AB situation).
• 3rd, compromise option is to
  • Treat -AA as AAA and NO_CONFLICT for increased usability.
  • Treat -AB as ABC and SOLVABLE_CONFLICT for safety. In this case, we would successfully auto-merge the change and return a merged_version: 'C', but mark it as SOLVABLE_CONFLICT according to the idea of solvable and non-solvable conflicts described in the RFC and this ticket. In this case, users would need to manually accept the auto-merged value in the upgrade flyout.

We decided to go with the 3rd option.

Having the 3rd option implemented, it might be useful to add a notification or a warning for the user in the upgrade UI that the base version is missing.

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)