Support S3 API for JetStream #5486
khalil-omer
started this conversation in
Ideas
Replies: 1 comment
-
We have some thoughts around virtual blocks, which is an approach that could use S3 to offload these when not needed locally. Timeframe would be 2.12 though.. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
JetStream requires a fast local block storage service. This is great for use cases like KV and for read heavy workflows. However, there are a number of products that are using S3 as the fundamental durability layer and treating disks as just caches, and this pattern could have value for a lot of NATS use cases.
Examples of products using this pattern are WarpStream (Kafka on S3), Neon (Postgres on S3), and Axiom (log analytics).
It could be useful to explore how NATS could potentially benefit from supporting this as a configuration option.
I imagine the basic flow as follows:
publish()
to NATS server as usual (no change to client libraries)This would be a pretty involved project, but it would unlock infinite and low cost stream scale.
Beta Was this translation helpful? Give feedback.
All reactions