Separate workflow manager and workflow engine, moving the latter to the target resources

[iparask]

The RP task manager in EnTK does two operations. The first is to manage the workflow and get state updates from RP, while the second is to operate on the workflow and submit tasks to RP. RP creates a tagging feature that allows to specify, not only placement decisions, but also dependencies between tasks.

I propose to fully utilize the tags RP will offer. Based on those tags, EnTK can submit the whole workflow in one submission to RP and just receive state updates.

This will in effect separate the management and the execution of the workflow.

Opinions?

[andre-merzky]

Yes, I fully agree, but... The workflow orchestration should not live in the RP agent scheduler, but should get it's own component.

[iparask]

The workflow orchestration should not live in the RP agent scheduler, but should get it's own component.

Do you mean a component that will know which tagged tasks should move to the scheduler inside RP?

[andre-merzky]

Do you mean a component that will know which tagged tasks should move to the scheduler inside RP?

Yes, indeed - only jobs eligible to run (i.e., jobs which have their dependencies resolved) should reach the scheduler.