NATS Jetstreams creation and management

[CodeTherapist]

I am interested in utilizing NATS as a messaging service to facilitate communication among multiple applications. One observation I've made is that when a consumer begins and attempts to establish a connection in order to consume a stream, this action fails due to the producer not having created the stream yet. This behavior has been documented in the following issue: #3935.

Regrettably, the NATS server lacks the capability to directly configure the streams via the nats.config file. While suggestions such as using nack or terraform-provider-jetstream have been put forth as potential solutions in issue #3935, these may not always be feasible options.

Given this context, I'm seeking guidance on the optimal approach for setting up a NATS server that possesses prior knowledge of the required streams. I'm particularly interested in hearing from individuals who have hands-on experience with managing streams in a production environment.

In terms of potential solutions, I've devised two proposals:

Proposal 1:

Configure the streams preemptively within the nats.config file. This configuration would occur only once. I am aware that implementing a migration system for the streams could introduce some complexities.

jetstream {
   store_dir = "/path/to/jetstream/store"
   max_mem = 1024MB
   
   stream {
      name: "my_stream"
      storage: file
      subjects: ["my.subject.*"]
      max_consumers: 10
   }
}

Proposal 2:

Enable the execution of a script during the server's startup phase, as indicated in the server.go file. This script would utilize the Stream Management API to configure and migrate the stream setup, all of this occurring prior to any other client establishing a connection.

streaminit: "/path/to/jetstream/setup-app"

I am eager to gather insights and thoughts regarding these proposals, especially from those who have encountered and resolved similar challenges in real-world scenarios. Your experiences and recommendations would be greatly appreciated.

[ripienaar]

There are a few reasons for the current API-first stance.

NATS is a multi tenant system and we support decentralised authorization where no credentials for the accounts are stored on the server. We therefor cannot just add the definitions as in your example as accounts can be managed outside of the server config. Managing accounts, their limits and properties just would not be compatible with also managing streams in this manner

NATS is often managed by a central team with users of the system being external - those users cannot access the configuration file, scripts or anything on the server.

So we take an API first approach where the client language APIs, the CLI, Terraform or GitHub Actions can be used to create a high quality automated resource management system. It's easy to trigger these systems as a setup container in your container scheduler or from another boot up process if you dont use containers

[CodeTherapist]

I greatly appreciate your prompt and detailed response. Your insights into the design principles of NATS as a multi-tenant system and its emphasis on decentralized authorization make a lot of sense to me. I fully understand the challenges that arise when considering the inclusion of stream definitions directly within the server configuration, especially in scenarios where account management and authorization are separate concerns.

Your explanation regarding the external users of the system not having direct access to server configuration files, scripts, or server-side components also clarifies the need for an API-first approach. This aligns with the practical considerations of managing a system where a central team handles the server while external users interact with the services.

In my specific use case, the configuration process remains straightforward. The developers utilizing the NATS service do not possess direct access to the configuration file. However, it's important to note that they shouldn't be obligated to familiarize themselves with the intricacies of the Jetstream management API. This consideration arises from the fact that the startup sequence of the applications, both consumers and producers, lacks consistency and cannot be enforced.

I'm glad you highlighted the value of employing client language APIs, the CLI, and automation tools like Terraform or GitHub Actions to facilitate resource management. Your suggestion to use a setup container within a container scheduler or as part of another boot-up process is particularly intriguing.

Could you kindly provide an illustrative example of how such a setup container and container scheduler could be utilized in conjunction with NATS? This would greatly aid my understanding of the practical implementation of the recommended approach. Is there somewhere an example within any repo?

Once again, thank you for your time and expertise. Your guidance is immensely valuable as I navigate the best practices for working with NATS in a multi-application environment.

[ripienaar]

Therer is a recent thread on slack where I added some basic examples and others also shared

https://natsio.slack.com/archives/CM3T6T7JQ/p1692811932746459