Raft.db increases infinitely and fills the disk

[PavelYadrov]

Overview of the Issue

Consul Version: 1.14.7

Recently we've found out that consul was unable to become alive with following error:

2024-10-22T07:30:11.652Z [ERROR] agent.server.raft: failed to append to logs: error="unable to store logs within log store, err: "write /consul/data/raft/raft.db: no space left on device""

For some reason, Raft.db started to grow from constant size (Around 800Mb) and then overloaded provided PV (3G)

We were able to make it work only after full PV cleanup on all nodes and deleting raft folders.

Then we've tried to tune raft logs compaction and overall logs size with following parameters

"raft_boltdb": {
"NoFreelistSync" : true
},
"raft_snapshot_interval": "5s",
"raft_snapshot_threshold": 8192,
"raft_trailing_logs": 5000

But this only reduced raft.db grow (it was around 10Mb per minute, currently we have around 3.5Mb per minute)

We understand that consul might be overloaded, metrics show 3-4 KV operations per second, but consul itself works just fine.

We need to understand how to handle this situation with Raft.db size size, because eventually it will exceed any PV size that we will provide to it