[release/8.0-staging] Re-try loading ENGINE keys with a non-NULL UI_METHOD

[github-actions]

Backport of #109706 to release/8.0-staging

/cc @vcsjones

Customer Impact

• Customer reported
• Found internally

Reported by a customer in #109243. Customers that use certain OpenSSL engines may encounter issues when using SafeEvpPKeyHandle.OpenPrivateKeyFromEngine or SafeEvpPKeyHandle.OpenPublicKeyFromEngine. These methods use ENGINE_load_private_key or ENGINE_load_public_key, respectively. One of the parameters to these methods is called ui_method. This method is used to inform an engine how it can prompt for user input or other interactivity, if needed, when loading a key.

Currently, NULL is passed as this parameter because we do not expect to show a user interface from this API, or any other kind of interactivity.

However, at least one engine, tpm2tss, will outright reject a NULL ui_method and fail to load the key.

This change accommodates engines that do not permit a null ui_method by trying again to open the key if it failed to open. The ui_method that we pass in is a stub one that does not perform any interactivity.

Customers are unable to work around this issue on their own.

Regression

• Yes
• No

Testing

This was manually validated. The issue is reproducible, and with the change, the error no longer occurs. Unit testing this is not feasible - it requires a particular native dependency set up and be installed (tpm2tss) as well as the corresponding OpenSSL engine. The machine must also have a functional TPM, which is not present in CI.

Risk

Low. This adds a fall-back mechanism. The way we were originally opening the key from an OpenSSL engine is still present. If it fails, we try again a different way to get it to load. Keys that were loading successfully already will continue to do so as the fallback path will never be taken. If the keys were failing to load, the worse case is they fail to load twice now.

IMPORTANT: If this backport is for a servicing release, please verify that:

• The PR target branch is release/X.0-staging, not release/X.0.

• If the change touches code that ships in a NuGet package, you have added the necessary package authoring and gotten it explicitly reviewed.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[vcsjones]

I am going to close this back port request for now as we do not believe it will resolve a customer's scenario. If we later receive confirmation from someone that this unblocks a scenario for them, we can open it back up.