[8.10](backport #3518) Fix: Secondary fallback is not used for agent with local agent

[mergify]

This is an automatic backport of pull request #3518 done by Mergify.

---

Mergify commands and options

More conditions and actions can be found in the documentation.

You can also trigger Mergify actions by commenting on this pull request:

• @Mergifyio refresh will re-evaluate the rules
• @Mergifyio rebase will rebase this PR on its base branch
• @Mergifyio update will merge the base branch into this PR
• @Mergifyio backport <destination> will backport this PR on <destination> branch

Additionally, on Mergify dashboard you can:

• look at your merge queues
• generate the Mergify configuration with the config editor.

Finally, you can contact us on https://mergify.com

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-10-09T10:41:14.213+0000

• Duration: 28 min 21 sec

Test stats 🧪

Test	Results
Failed	0
Passed	6317
Skipped	55
Total	6372

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

• /package : Generate the packages.

• run integration tests : Run the Elastic Agent Integration tests.

• run end-to-end tests : Generate the packages and run the E2E Tests.

• run elasticsearch-ci/docs : Re-trigger the docs validation. (use unformatted text in the comment!)

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	98.765% (80/81)	👍
Files	67.944% (195/287)	👍 0.348
Classes	66.234% (357/539)	👍
Methods	53.125% (1122/2112)	👍 0.117
Lines	38.575% (12749/33050)	👍 0.043
Conditionals	100.0% (0/0)	💚

[cmacknz]

I believe this has missed the 8.10.3 BC since it is still open.

@jlind23 did we want this in 8.10? It looks like it was targeted to 8.11 originally, happy to just close this one since this a feature not a bug fix.

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[pierrehilbert]

Just saw your comment @cmacknz.
I tend to agree with you, the timing is too short to get it in 8.10.3.
However a 8.10.4 will probably be released so we could still get it in 8.10 if we want it.

[mergify]

This pull request has not been merged yet. Could you please review and merge it @michalpristas? 🙏

[jlind23]

@pierrehilbert let's get it in as soon as possible though.

[pierrehilbert]

@lucabelluccini as we discussed this this morning.

[lucabelluccini]

What is the behavior this PR is fixing? Is it the ability of Elastic AGent to use Fleet Server as 2nd fallback for GPG key?
If it is the case, having it on a release soon allows more users to use this as fallback in case the GPG will be ever rotated on early version (8.10.x).
But my opinion is it's not critical to ship in 8.10.x - I see it as a nice to have. It must be in 8.11.0.

Can also anyone confirm:

• Try with embedded GPG Key
• If success, stop
• If fails (cannot verify)
  • Log what happened as WARN or INFO
  • Try with public GPG Key
  • If success, stop
  • If fails (cannot verify or cannot download)
    • Log what happened as WARN or INFO
    • Try with Fleet Server GPG Key
    • If success, stop
    • If fails (cannot verify or cannot access/download)
      • Log what happened as WARN or INFO
      • Fatal, stop

[michalpristas]

@lucabelluccini the scenario you described is correct.
8.11 was already merged

[pchila]

@michalpristas could you please check that this change (also for 8.11 and main) did not reintroduce instability for fleet bootstrap on cloud ? I am looking at the test results of removing the 8.10.2 stack version pinning #3563 (review) and it does not look encouraging...

Could you please test ?

Edit: me and @michalpristas had a look on main and this does not reintroduce the bootstrap mode we saw last week

[michalpristas]

discussed with @pierrehilbert we will wait with merge