Spike: v4 workflow refactor

[ferglor]

This is a spike aiming to reduce the complexity around the existing flows in our codebase

From a design perspective, we have quite a simple concept of having different pipelines read from different sources, and delivering results to different destinations. From a code perspective, we implement this concept in a needlessly complex way, across too many components.

In its current state, I think our implementation is needlessly complex because:

• We have generic types, but only ever use the generic types for UpkeepPayload slices in the production code
• We use an observer to apply pre processors, check upkeeps, and run postprocessors (Why can't the workflows run themselves?)
• We use a time ticker concept to supply payloads and trigger the observer (Again, why do we need a separate type to supply payloads and trigger a run of the observer?)

In this spike:

• We are removing some generic types (may not be needed in the final set of changes, this was just a starting point needed to remove errors and warnings in my IDE)
• We are deprecating the time ticker and observer concepts
• We are implementing a single pipeline type, responsible for running the pipeline (preprocessor, checkUpkeeps, postprocessor) on a configurable interval
• We are implementing workflow providers for each workflow (conditional proposal/final, log trigger/retry, recover proposal/final), which are each responsible for:
  • Providing payloads to the pipeline
  • Providing preprocessors to the pipeline
  • Providing a runner to the pipeline
  • Providing posprocessors to the pipeline

The overall goal here is to reduce the complexity and number of dependencies among different entiities, and have a much more procedural approach to workflows, with a reasonable amount of code reuse (i.e. the workflow pipeline is shared among the workflow providers, in that each workflow provider creates an instance of the worklow pipline, passing themselves as workflow providers for the pipeline to use)