Pipeline Events

[chubei]

Pipeline Events

Problem

In the current pipeline, the message flow follows the DAG pattern of the computation graph. While this is convenient and easy to understand, it's limiting in the sense that upstream nodes can't react to the output of downstream nodes. While in general upstream nodes doesn't need access to downstream nodes' data, it may need to have access to some "metadata", which we call the "pipeline events".

Visualization of current message flow.

Solution

We introduce a new component in the pipeline, the Hub. The computation nodes will send events to the Hub, which gets broadcasted to all nodes.

Visualization of proposed message flow.

Alternative

The nodes can also send both data and events to the Hub, and the Hub will be responsible for dispatching the data to downstream nodes.

The pro of this approach is the unified data and event flow, which is easier to understand and maintain. The con is that the change is bigger than the proposed solution.

Visualization of alternative message flow.