Skip to content

Commit

Permalink
Add Aviso+Mirror how-to instructions
Browse files Browse the repository at this point in the history
Re ECFLOW-1931
  • Loading branch information
marcosbento committed Jun 18, 2024
1 parent 655043e commit 97113ae
Show file tree
Hide file tree
Showing 3 changed files with 137 additions and 3 deletions.
70 changes: 70 additions & 0 deletions docs/ug/cookbook/how_to_mirror_a_remote_task_status.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
.. _how_to_mirror_a_remote_task_status:

How to mirror a remote Task status?
-----------------------------------

The following instructions describe the necessary steps to use the ability to
synchronise the status of a :term:`task` between a `remote` ecFlow server
and a `local` ecFlow server. The `remote` server is responsible for actually
executing the task, while the `local` server synchronizes the status of a
particular task, in order to trigger the execution of dependent tasks.

The deployment of this feature has the following requirements:
- `local` ecFlow 5.13+, to host the dependent task(s)
- `remote` ecFlow 5+, to host the remote task
- Authentication credentials, stored as JSON file

Setup the ecFlow Server
^^^^^^^^^^^^^^^^^^^^^^^

Deploy the `remote` ecFlow server, as per the :ref:`regular instructions<starting_the_ecflow_server>`
and specifying username/password access configuration.

Deploy the Authentication credentials so that the JSON file is accessible
to the `local` ecFlow server. Launch the server.

Consider using SSL encription for the communication between ecFlow server and
client.


Define the Suite with a `mirrored` Task
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

On the :term:`suite` definition file, create a :term:`task` (which will mirror
the remote task status) and assign it a :term:`mirror` attribute. The mirror
attribute must specify the path of the remote node, and the remote server
connection details. Using the mirrored task, other tasks can use triggers to
execute when the remote task becomes complete.

Follow the recommended practice of defining ecFlow :term:`variables<variable>`
at :term:`suite` level:

- ECF_MIRROR_REMOTE_HOST
- ECF_MIRROR_REMOTE_PORT
- ECF_MIRROR_REMOTE_POLLING
- ECF_MIRROR_REMOTE_AUTH

**Example 1**: Example suite definition with mirror attributes

.. code-block:: shell
suite s
edit ECF_MIRROR_REMOTE_HOST 'remotehost'
edit ECF_MIRROR_REMOTE_PORT '3456'
edit ECF_MIRROR_REMOTE_POLLING '120'
edit ECF_MIRROR_REMOTE_AUTH '/path/to/mirror.auth'
family f
task Task
mirror --name A --remote_path /s1/f1/t1 --remote_host %ECF_MIRROR_REMOTE_HOST% --remote_port %ECF_MIRROR_REMOTE_PORT% --polling %ECF_MIRROR_REMOTE_POLLING% --ssl
task Dependent
trigger Task == complete
endfamily
endsuite
Deploy the Suite with a `mirrored` Task
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Load the suite definition containing the :term:`mirror` attribute.

Whenever the ecFlow server is in `RUNNING` state, the synchronisation between
`remote` and `local` servers is enabled automatically.
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
.. _how_to_trigger_a_task_based_on_aviso_notification:

How to trigger a Task based on Aviso notification?
--------------------------------------------------

The following instructions describe the necessary steps to use the ability to
trigger a :term:`task` based on an Aviso notification. The intended task must
be assigned an :term:`aviso` attribute, which provides the configuration to
periodically poll the Aviso server, and execution only happens when the
a matching Aviso notification is processed.

The deployment of this feature has the following requirements:
- ecFlow 5.13+, to host the Aviso dependent task(s)
- Aviso event listener schema
- Authentication credentials, stored as JSON file

Setup the ecFlow Server
^^^^^^^^^^^^^^^^^^^^^^^

Deploy the Authentication credentials and Aviso event listener schema files so
that these files are accessible to the ecFlow server. Launch the server, as per
the :ref:`regular instructions<starting_the_ecflow_server>`.

Define a Suite with an `Aviso` dependent Task
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

On the :term:`suite` definition file, create a :term:`task` and assign it an
:term:`aviso` attribute.

The :term:`aviso` attribute must specify the base URL to contact the Aviso server,
and the remote server connection details. Using the mirrored task, other tasks can use triggers to execute
when the remote task becomes complete.

Follow the recommended practice of defining ecFlow :term:`variables<variable>` at :term:`suite` level:

- ECF_AVISO_URL
- ECF_AVISO_SCHEMA
- ECF_AVISO_POLLING
- ECF_AVISO_AUTH

**Example 1**: Example suite definition with mirror attributes

.. code-block:: shell
suite s
edit ECF_AVISO_URL 'http://aviso:2379'
edit ECF_AVISO_SCHEMA '/path/to/event_listener_schema.json'
edit ECF_AVISO_POLLING '120'
edit ECF_AVISO_AUTH '/path/to/aviso.auth'
family f
task Task
aviso --name A --listener '{ "event": "mars", "request": { "step": 0 } }' --url %ECF_AVISO_URL% --schema %ECF_AVISO_SCHEMA% --auth %ECF_AVISO_AUTH% --polling %ECF_AVISO_POLLING%
endfamily
endsuite
Define a Suite with an `Aviso` dependent Task
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Load the suite definition containing the :term:`aviso` attribute.

Whenever a task assigned with an :term:`aviso` attribute is (re)queued,
the Aviso will start to be periodically poll. The task is held from execution
until a notification matching the provided listener is found.
Once the task is allowed to execute the periodic polling is stopped, until the
task is requeued again.
5 changes: 2 additions & 3 deletions docs/ug/cookbook/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,6 @@
Cookbook
============


This cookbook will provide real examples of using the ecFlow :ref:`python_api`.

**Contents**
Expand Down Expand Up @@ -38,5 +37,5 @@ This cookbook will provide real examples of using the ecFlow :ref:`python_api`.
native_python_task
one_script_do_it_all
python_client_for_fault_tolerance


how_to_mirror_a_remote_task_status
how_to_trigger_a_task_based_on_aviso_notification

0 comments on commit 97113ae

Please sign in to comment.