msgBlock + msgFromBuf (filestore.go) consumes a lot of memory

[nuttert]

Observed behavior

Using pprof I see that nats-server consumes RAM more that expected, because my streams ≦ 1 GB:

Here is an example configuration of a stream:

I also do not see any problems with the disc IO(340 MiB is a maximum):

I do not see slow consumers:

Expected behavior

I expect that nats consumes several GB of memory instead of 40GB. I believe that it's not a leak because it does not go over ~40GB usually.
Probably the problem was triggered after I got started to use FIle storage instead of memory.

Server and client version

NATS server version 2.10.14
client github.com/nats-io/nats.go v1.37.0

Host environment

Memory: 128 GB
CPU: 16 cores

without container, 22.04.1-Ubuntu.

Steps to reproduce

No response

[neilalexander]

Are all the streams a similar configuration with the same retention policy (limits)?

Do your streams have a large number of interior deletes or do messages only get deleted when they age out by limits?

Also, 2.10.14 is a bit behind at this point, would help if you can see if you can reproduce with 2.10.22.

[nuttert]

Are all the streams a similar configuration with the same retention policy (limits)?

Yes

do messages only get deleted when they age out by limits?

Yes, clients do not delete messages

Also, 2.10.14 is a bit behind at this point, would help if you can see if you can reproduce with 2.10.22.

Okay, I will try.

Also, just to be clear, I don't use Kubernetes or any containers, I just run Ubuntu 22.0