EnTK next features and requirements.

[iparask]

This is a discussion about the new features and capabilities offered by EnTK. The document that has all the feature requests, as well as their requirements, is here

[iparask]

@mturilli, @andre-merzky, @lee212, if my memory serves me, the streaming request for EnTK is mainly restricted with ADIOS. My understanding is that this is a method to pass quickly data between processes that are executing.

For example, we have two simulations, A and B, which are executing and A produces some intermediate data that B will use as an intermediate input. Does this summarize the request?

If that is correct, does the common sandbox between tasks cover it? I believe I'm missing something.

[iparask]

I'm working on REQs- 3 and 4. I propose the following:

1. Introduce new public methods in the Task Manager:
   a. add_resources and remove_resources: The purpose of these methods is the allow the addition of new resources and removal of lost resources. In the case of RP they can be like:

def add_resources(self, pilot):
    self._tmgr.add_pilot(pilot)

def remove_resources(self):
    pilot = self._tmgr.list_pilots()
    self._tmgr.remove_pilot(pilot_ids=pilot)

b. update_resources: This methods works under the assumption that only one set of resources is available at any given point in time and can look like:

def update_resource(self, pilot):
    curr_pilot = self._tmgr.list_pilots()
    if curr_pilot:
        self._tmgr.remove_pilots(pilot_ids=curr_pilot)
    self._tmgr.add_pilot(pilot)

2. Resource manager _submit_resource_request will return the latest pilot. As a result, the AppManager will have access to the resources object and be able to communicate it to the Task Manager.
3. The AppManager _run_workflow method will request new resources when the current ones fail, or get canceled, inform the task manager of the change, and reset the workflow.

[iparask]

I'm trying to prototype solution b, and I realized that the AppManager, resource manager, and Task Manager live in different processes/threads. So, I'm bringing this image back:

We have 2 processes, one that executes the AppManager (same in the figure) and one that runs the Task Manager (tmgrProc in the figure). Furthermore, tmgrProc has a thread called Thread-1 in the figure. This thread launches and holds RP's TMGR. Both proposed solutions need to call methods from objects that live in different processes/threads, which do not nessecerily have access to.

Implementing these methods as callbacks provide a solution, but I think they add additional complexity to the design of EnTK.

Does anyone have any thoughts or ideas?

[iparask]

REQ-7

This requirement asks for the non-blocking execution of a set of pipelines.

Pipeline, Stage changes:

Stages

Although there are no changes in the user-level API for these three entities. However, during non-blocking execution, new tasks can be added to an active stage. Currently, when a stage is to be executed EnTK creates a set of its tasks, after resolving any dependencies. If the stage is active and a new task is added, we want to include the task to the current workload and push it to the RTS for execution. We can introduce a sticky bit in the set where tasks are in a stage. This sticky bit will be set when a task is included in the workload. The Task Manager can periodically check the sticky bits of all tasks in a stage to determine if a new task has arrived. If it finds a new task, it pushes it immediately to the runtime system. If all tasks of a stage have finished execution the task transitions to the done state and no more tasks are allowed to be added.

Pipelines

1. Introduce pipeline.terminate() method to move a pipeline to a final state. This will allow the appman to finalize the execution of a workflow.

Application Manager API changes:

• appman.submit_pipelines(set([p])): Adds additional pipelines for execution. This is similar to updating the workflow after another workflow has finished. There is already a method in the task manager that “resets” the execution of a workflow. We can extend that method to support the case, where more pipelines are added.
• appman.run(blocking=False): We introduce a blocking parameter in the run method. Execution of the workflow, essentially what is now part of the method, moves to a separate thread. The tread is responsible for executing the workflow. If blocking is True, then the method waits until the thread has finished and joins it, otherwise, it just returns.
• appman should be able to transition pipelines to a final stage when blocking=False

Ideas?