Refactor workload control loop

[inf17101]

Description

The agent/src/workload/workload_control_loop.rs file has a little bit too much responsibilities. We can decouple more and simplify the unit tests here, when:

1. Extracting the current concrete operations "create, update, delete, retry, restart) into own sub modules (perhaps with introducing a trait to decouple the knowledge about the concrete operations from the control loop itself)
2. Refactoring the WorkloadControlLoop to only be responsible for receiving the workload states of the state checker and the workload commands and delegating the concrete operation to the specific module from point 1.
3. The tests can then be simplified to only verify that when the WorkloadControlLoop receives a workload state or a workload command that it calls the appropriate module (just asserting the mock) or checking that the ControlLoopState variable contains the expected field values.

Goals

The WorkloadControlLoop shall be decoupled from the concrete operations it executes and unit tests shall be simplified.

Final result

Summary

To be filled when the final solution is sketched.

Tasks

• Task 1
• Task 2
• ...

[inf17101]

With introducing the config files mount for workload within #426 (Issue #393), the logic for cleaning up the config_files subfolder upon an update or delete must be implemented into the WorkloadControlLoop as well. This is due to the fact that the operations are already queued because of the async channel within the WorkloadControlLoop to prevent side effects.

In addition, the ControlInterface creation happens outside of the control loop, in the RuntimeFacade for create and resume and in the workload.rs for the update operation. This means the ControlInterface creation decoupled in time from the actual atomic and queued workload operations. This leads to the unsightly side effect that when doing cleanup of subfolders of the workload after the delete part of an update that it must be checked with conditionals "what the control loop is allowed to clean up". Because it might be the case that the control interface remains at the same location for an update and then cleaning up the whole workload folder leads to unwanted deletion of the control interface. The problem is that the control interface belongs actually to the atomic update "plan" within the control loop, because then the steps can be performed in the correct order to avoid the "cleanup checks". When refactoring the control loop, consider to move the creation of the control interface into the correct "create", "update" and "delete" plan within the control loop eliminating unnecessary checks afterwards.

The rows signal the timeline.

workload.rs	workload_control_loop.rs::update operation
Exchange or keep control interface on update
Send update command to the control loop
	Delete old workload and stop state checker
	Unnecessary cleanup checks (control interface, config files or all?)
	Cleanup steps itself
	Create new workload

To simplify the code, group together what belongs together and fix the order of the update plan itself:

workload.rs	workload_control_loop.rs::update operation
Send update command to the loop
	Delete old workload and stop state checker
	Delete control interface
	Delete the whole workload folder
	Create a new control interface
	Create new workload

Grouping thins belonging together reduces complexity and if a reader wants to see all the steps of the upate plan it is all in one place.

[inf17101]

In addition, @krucod3 suggested to let the functions return an error, so that it can be logged once in the match statement distinguishing between the workload commands.

[inf17101]

There are also state machine crates: https://docs.rs/rust-fsm/latest/rust_fsm/ that might help. Or we implement our own state machine to refactor this control loop. Also a behavior tree implementation might help, which is more efficient to define as a state machine: https://github.com/msakuta/rusty-behavior-tree-lite