Scheduled upgrade fails with error parsing version "" if agent restarts before the upgrade starts

[AndersonQ]

• Version: 8.12/main
• Operating System: all
• Discuss Forum URL: N/A
• Steps to Reproduce:
  • schedule and upgrade on fleet for 2 min form now
  • check the agent report the upgrade is scheduled
  • restart the agent
  • wait the upgrade to start
  • it'll fail:

# elastic-agent status                                                                                                 

┌─ fleet
│  └─ status: (HEALTHY) Connected
├─ elastic-agent
│  └─ status: (HEALTHY) Running
└─ upgrade_details
   ├─ target_version:
   ├─ state: UPG_FAILED
   ├─ action_id: 9c06cd82-50e5-41bc-8a68-ed73b55c1912
   └─ metadata
      ├─ failed_state: UPG_DOWNLOADING
      └─ error_msg: could not download artifact: error parsing version "": version string does not match expected format

The issue seems to be on how the actions are loaded from disk. Using the patch patch.txt we can see the test will fail because the upgrade action version isn't loaded.
If you stop the test and check the store file, it contains all the data, but it sin't loaded correctly:

• state.yml:

action_queue:
- action_id: test
  type: UPGRADE
  start_time: "2023-12-14T08:09:59Z"
  version: 1.2.3
  source_uri: https://example.com
  retry_attempt: 1
- action_id: abc123
  type: POLICY_CHANGE
  policy:
    hello: world

Bug root cause

TL;DR: we deserialise the action from one schema (the fleetapi.FleetAction) and serialise a different schema (our concrete action types, like the fleetapi.ActionUpgrade. When reading from disk, we loose data, such as the version from the upgrade actions.

Proposed fix

• use only JSON to serialise/deserialise the actions
• make all fleetapi.ActionTYPE models to match the schema we receive from fleet. Add any action specific properties nested under data
• migrate the current state store in YAML to a new state store using JSON
  • properly deserialise the old store to do not lose data during the migration

Fix implementation phases (using a feature branch):

• Create a test for the migration from the old action_store to the current state_store using a golden file. The current test serialises the action_store on the fly, which renders the test pointless if there is any changes to the serialisation process. Fix action store -> state store migration tests #4235
  • Fix action store -> state store migration tests #4235
• Refactor state store #4253
• change the fleetapi.ActionTYPE to match the fleet-server model by adding the specific action properties nested under a data property. Match fleet server action model and use on JSON for serialisation #4240
• write the migration from the current state_store in YAML to a new one in JSON Add action and state stores migrations #4305
• merge the feature branch to main Fix and refactor the state store #4441
• integration test: Add an upgrade integration test upgrading to the PR build #4560
• QA test: QA test: State Store migrations #4962

[elasticmachine]

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

[cmacknz]

@pierrehilbert we should get this one fixed, likely it leads to failed upgrades or agents stuck in updating.

[pierrehilbert]

Agree, I already added this to current sprint but didn't assign this yet.
Need to see what we can postpone to take care of it.

[elasticmachine]

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

[elasticmachine]

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)