Operation Retry

[peterbroadhurst]

The design of FireFly prioirtizes idempotent "at least once" semantics wherever possible, fitting with the modern approaches taken in the underlying technologies it integrates - like blockchain, REST, Kafka, IPFS etc.

We define an "operation" in the database every time an action is instigated by a node in the network.
These objects do not propagate - they are that member's local record of the part of the overall "transaction" they submitted.

Multiple operations roll up into different types of "transaction", and coordinating these actions and events into a programming model is a core function of FireFly (per the "ping pong" diagram #5)

Some examples:

• Broadcast requires an IPFS operation, and a Blockchain (eth/fabric) operation
• Private pinned send requires a set of DX operations to each member, and a Blockchain operation
• Token pool creation is the first higher level transaction, orchestrating:
  • A token operation to create, or find details about, a token on the chain
  • A broadcast transaction, to broadcast that data to the whole network

The reason these operations are stored in the database, is so that if they fail, they can be retried.
Each connector should have a level of retry built in, but it is impractical to retry forever. Sometimes failures require.

The retry API/UI mechanisms have not been implemented (you can just query today), and there are some considerations coming out that need to be factored in when it is.

This issue should be used for the architectural discussion on how operations evolve into safely retryable item, as started in some previous PRs - notably #211 (comment)

Also this item on sophisticated requirements for policy-driven retry in public chain scenarios:
hyperledger/firefly-ethconnect#149 (comment)

[peterbroadhurst]

Note should be merged with #400

[awrichar]

Duplicate of #400