State management for JS apps

[chubei]

State management for JS apps

Connector

As revealed in the discussion, our connectors as they are now are actually stateful, only that the state is in a very limited form: the OpIdentifier.

Now that we've discarded the need for the OpIdentifier to be Ord, we can actually rebrand it as the connector's "state", and allow connectors to send arbitrary data to Dozer.

Persisting and restoring will be handled as it is now.

UDF

We decided that functions can be stateful, and will add persist method to Expression.

We'll provide a KV store for the the JavaScript UDF, and the store will be persisted in persist.

Lambda

App restart

To set the context, lambda connects to app and is subject to rollback problem upon app restart.

There are two options:

1. Deliver real-time events to lambdas, and ask the lambda author to handle rollback.
2. Only deliver events are they are persisted.

We'll make this configurable and default to real-time.

Lambda restart

Lambda runtime will maintain the state of which messages have been delivered. A message is considered as delivered when the lambda function completes, including the awaiting for async lambdas. Delivered messages will not be delivered again.