JetStream failed to store a msg on stream '$G > : error opening msg block file [""]: open : no such file or directory

[anpatel1996]

Hello Team,

We are using NATS-Jetstream 2.9.14 with File Storage. Max size 2.5 GB.
System was running for ~30 days with minimal traffic on NATS.
We had started with around ~50 streams. Meaning we had close to 50 directories in jetstream/$G/streams/. After some days we are encountering JetStream failed to store a msg on stream '$G > : error opening msg block file [""]: open : no such file or directory error and on checking persistent directory(jetstream/$G/streams), I see only 5 stream's directory present. Somehow other directories are deleted. But I can see all the streams on nats stream info.
What could be the reason for deletion of directories?? Does jetstream deletes inactive stream's directories based on some configuration?? How can we find out the reason for being in this situation??

This is the typical example of stream config and consumer config

? Select a Stream stream-1
Information for Stream stream-1 created 2024-08-07 19:07:11

          Subjects: sub-1
          Replicas: 1
           Storage: File

Options:

         Retention: Limits
   Acknowledgments: true
    Discard Policy: Old
  Duplicate Window: 2m0s
 Allows Msg Delete: true
      Allows Purge: true
    Allows Rollups: false

Limits:

  Maximum Messages: 25,000

Maximum Per Subject: unlimited
Maximum Bytes: unlimited
Maximum Age: 2m0s
Maximum Message Size: 4.0 MiB
Maximum Consumers: unlimited

State:

          Messages: 0
             Bytes: 0 B
    First Sequence: 48
     Last Sequence: 47 @ 2024-08-13 13:09:27
  Active Consumers: 2

this is one of the consumer's configs

$ nats consumer info
? Select a Stream stream-1
? Select a Consumer stream-1-consumer-1
Information for Consumer stream-1 > stream-1-consumer-1 created 2024-08-07T19:07:24+05:30

Configuration:

                Name: stream-1-consumer-1
    Delivery Subject: stream-1-consumer-1-sub
      Filter Subject: sub-1
      Deliver Policy: All
          Ack Policy: Explicit
            Ack Wait: 30.00s
       Replay Policy: Instant
     Max Ack Pending: 1
        Flow Control: false

State:

Last Delivered Message: Consumer sequence: 47 Stream sequence: 47 Last delivery: 43d2h32m37s ago
Acknowledgment Floor: Consumer sequence: 47 Stream sequence: 47 Last Ack: 43d2h32m37s ago
Outstanding Acks: 0 out of maximum 1
Redelivered Messages: 0
Unprocessed Messages: 0
Active Interest: Active

[neilalexander]

NATS shouldn't be deleting folders from the disk unless the streams themselves were deleted or moved off that node (due to a cluster resize/move). Can I ask why you are still on such an old version?

[anpatel1996]

Thanks for the response Neil.
We understand that we are currently on an older version of NATS and are planning to upgrade it soon.
That said, would like to understand if the version we're on could still have any impact on the deletion of persistent directories. Specifically, could the version affect the handling of these directories even if no manual deletion was performed?

[anpatel1996]

Hi @neilalexander we have upgraded to newest version 2.10.21 and we faced same issue. I am mostly suspecting it because of inactivity on those streams. Can you suggest some configuration or some way to avoid it??

[anpatel1996]

Logic behind it was faster access but directories just disappearing is not okay right?

[neilalexander]

Yes, it's definitely not OK for filestore if directories/files disappear from under the server. We do in-memory caching of filestore blocks to help speed up accesses, and if you are worried about how much space on disk you'll take up, you should use stream limits.

[anpatel1996]

Atleast we don't do any cleanup at all on our end. Just wanted to know any logic behind jetstream could be responsible for it?
We are seeing this symptom https://groups.google.com/g/natsio/c/f_8KRoVZi0Q without we implementing anything.

[neilalexander]

Many Linux distributions have tools, such as tmpwatch, tmpreaper and tmpfiles.d, enabled by default to automatically delete files from tmpfs after they reach a certain age. You should never rely on RAM disks being anything other than temporary.

This is not JetStream behaviour and the post you linked from 2016 is pre-JetStream (it is for the deprecated NATS Streaming).

[neilalexander]

I should add, we do have a memory store option for streams if you really want to keep the data in memory instead of on disk, with the obvious caveat that the data goes away when the process restarts, but pointing the file store to tmpfs is not a supported/endorsed configuration.