How to distirbute load from one stream to all JetStream cluster nodes?

[Envek]

We have an app with one main WorkQueuePolicy stream for communication between components, stream is located on a 3-node JetStream cluster.

However, there is very high load on one of the nodes, where all publishers and subscribers are, while other nodes are idling (basically only getting stream replicated to them).

Is there a way to distribute load from a single stream between cluster nodes transparently to producers and consumers?

This problem seems to be very close to the one discussed at #3554

One workaround solution to this problem would be like the one proposed by the author of aforementioned #3554:

1. create 3 streams (by the number of nodes in the cluster)
2. move each stream to a different node (via placement tags or maybe via forcing leader re-election)
3. make producers to publish work to a different stream each time (via round-robin, hashing, or random, it probably doesn't matter)
4. make consumers to subscribe to all streams at once and somehow aggregate and process messages

This approach makes application code significantly more complex (especially due to need to implement message aggregation from many streams in consumers), leaks cluster topology knowledge to clients, and makes cluster scaling harder (especially downscaling).

Is there a simpler way to spread the load?

And thank you for the NATS, it is awesome!

[ripienaar]

Consumers are bound to a single node and writes into a stream also - with some additional constraints for WQ and Interest based streams.

So you're about at a combination of the hardest parts to make horizontally scalable.

You can simplify some of the 4 steps a bit by a combination of server side subject mapping so your producers dont need to be aware of the partitions. Could also have one limits stream that sources into n other WQ streams. This might also simplify things a tad but really with server mappings its best.

On the consumption side its not too bad to enumerate the streams programatically subscribing to each

Agree this is not ideal but thats what we have today

[derekcollison]

Consumers leaders do not have to be on the same server as the stream leader, so if that seems to be the case you could move one of the leaders.