Processing pipelines?

[sskserk]

Hi Nats developers,

do you have any plans to introduce data processing pipelines in your product?
In fact you already provide all required event driven architecture capabilities. Messaging, events, scrolling over the history, distributed fail safe storage. The next step could be a breakthrough kick if it is the data processing pipelines.

Simple lambda like embeddable packages with processing functions that connect to a topic or queue, and publish results to the next queuing stage could significantly increase usage of your product.

Any ideas on that?

[Kinrany]

What would it look like if one wanted to implement this on top of NATS?

[sskserk]

Maybe I'm trying to compare NATS with Kafka + Kafka Streams here. I'm mentioning a potential way to process data while it flying over NATS. At this moment it could already be processed, and as the result consumers or subscribers will receive already cooked and ready to eat data. That feature is the data processing pipelines. NATS already stores data, it is an event broker. Why not to implement an adhoc logic within?

[derekcollison]

Yes we have been looking into this on several fronts. One is through the project Benthos.
When we have more to share formally we will.

[sskserk]

@derekcollison hm. That's a good idea. Indeed, it always makes to integrate something already mature enough. Do you have any predictions a possible release date? Even, beta would be already good enough!

[derekcollison]

We don't, however NATS has become a first class citizen in Benthos, so you can explore today.

[sskserk]

@derekcollison are you saying that NATS could be accompanied with the Benthos? It is not Benthos will be built in into NATS. Is it the NATS became a member of the Benthos communiy and due to this reason smoothly integrated with it?

[derekcollison]

Yes NATS is a first class citizen now in Benthos as a source and a sync. We have some other things in the works but too early.