Publish multiple events or none

[walling]

I’m interested if there’s a pattern in Liftbridge, where you can publish multiple events or none at all (failing). This would be useful for domain driven development aggregates, where sometimes you need to make sure that all events for a given aggregate are stored (transaction boundary).

I was thinking that it might be possible to support through an extra stream, where you publish a list of all events in a single message. This message is read by a command handler that (idempotently) publishes each event to the aggregate stream. If this fails half way through, the message can be repeated until successful, because of idempotent publishing. This could potentially be implemented using the expectedOffset option.

However, this solution seems convoluted. Is there a simpler or more direct way?

I appreciate that Liftbridge strives for a simple architecture and design, so it might be overkill to add complex transaction capabilities into Liftbridge (like a PublishMultipleMessages endpoint or even more generic transactions patterns). At least, if this pattern can be supported through a client library.

As a side note, I’m interested in a comparison of features between Liftbridge and EventStoreDB. Should I post an issue for it?

[tylertreat]

I think transactions are certainly on the table for Liftbridge, though not currently on the roadmap (this can change). It's a fairly large/complex feature and consumer groups is another large/complex feature that is getting implemented first. I would need to think through implementation for transactions, though we could look to how Kafka does them.

As a side note, I’m interested in a comparison of features between Liftbridge and EventStoreDB. Should I post an issue for it?

I'm not familiar with EventStoreDB, so I'm afraid I can't provide much insight there, though perhaps someone else can weigh in.

[walling]

Thanks for your answer. I appreciate it. Looking forward to the consumer groups feature as well.

Have you considered a variant, like small transactions / unit-of-work model, where a single message containing multiple guards and actions is atomically applied? E.g. like etcd does it.

However, I see benefit in having long-running transaction support as outlined in the linked Kafka document. This can scale to a large number messages per transaction.

[tylertreat]

Have you considered a variant, like small transactions / unit-of-work model, where a single message containing multiple guards and actions is atomically applied? E.g. like etcd does it.

I think this is interesting and probably much more tractable as a short-term solution. What would be helpful is if you could sketch out a possible API if you have something in mind. Then we can open up the API proposal for discussion.

[walling]

Sorry, I've been a bit inactive in this thread.

I have given it a bit of thought. Being able to publish multiple messages and set the cursor atomically would support my use-case. Updating the cursor must be included in the transaction, otherwise it could lead to repeated publishes (by lacking acknowledgement from the stream consumer/producer). Idempotent publishes (i.e. guarding on a message offset) would also be beneficial. I could probably sketch out a API suggestion.

However, I'm not sure how to validate whether it's a good model to support other use-cases that need transactions. E.g. what kind of read/write operations should be supported by the model.

[tylertreat]

Updating the cursor must be included in the transaction

This makes things a bit more complicated since cursors are themselves a separate stream that gets published to in order to set/update a cursor. Basically this requires atomic publishes across streams/partitions, which I think would put us back into the territory of Kafka-like transactions.

[walling]

This makes things a bit more complicated since cursors are themselves a separate stream that gets published to in order to set/update a cursor. Basically this requires atomic publishes across streams/partitions, which I think would put us back into the territory of Kafka-like transactions.

Yes, I believe it's fine to push this issue back a bit and do it properly.