cluster node crash on taking raft snapshot

[zajakala]

On a busy cluster (a lot of simultaneously connecting mqtt clients) I occasionally run into the following error

fatal error: concurrent map iteration and map write

goroutine 61 [running]:
reflect.mapiternext(0x4d7629?)
        /usr/local/go/src/runtime/map.go:1397 +0x13
reflect.(*MapIter).Next(0xc0467bfc40?)
        /usr/local/go/src/reflect/value.go:2005 +0x74
encoding/gob.(*Encoder).encodeMap(0xc00042c000, 0xc2ae01b740?, {0x119b0a0?, 0xc0004cfa00?, 0xc1e7403830?}, 0x139e6b0, 0xc2a8a01dc0, 0x0, 0x0)
        /usr/local/go/src/encoding/gob/encode.go:373 +0x545
encoding/gob.encOpFor.func3(0x119b0a0?, 0xc0467bfc00, {0x119b0a0?, 0xc0004cfa00?, 0x0?})
        /usr/local/go/src/encoding/gob/encode.go:536 +0x12c
encoding/gob.(*Encoder).encodeSingle(0x10ea760?, 0xc0004cfa00?, 0x14f7198?, {0x119b0a0?, 0xc0004cfa00?, 0x407f4b?})
        /usr/local/go/src/encoding/gob/encode.go:301 +0x1fc
encoding/gob.(*Encoder).encode(0xc00042c000, 0xc00042c038, {0x10ea760?, 0xc0004cfa00?, 0xc0467bfa80?}, 0xc0467bfa80)
        /usr/local/go/src/encoding/gob/encode.go:668 +0x1d3
encoding/gob.(*Encoder).EncodeValue(0xc00042c000, {0x10ea760?, 0xc0004cfa00?, 0x38?})
        /usr/local/go/src/encoding/gob/encoder.go:251 +0x405
encoding/gob.(*Encoder).Encode(0x3?, {0x10ea760?, 0xc0004cfa00?})
        /usr/local/go/src/encoding/gob/encoder.go:176 +0x75
github.com/wind-c/comqtt/v2/cluster/raft/hashicorp.(*Fsm).Persist(0xc0004d64c0, {0x14e3440, 0xc000426b40})
        /app/cluster/raft/hashicorp/fsm.go:88 +0x1f0
github.com/hashicorp/raft.(*Raft).takeSnapshot(0xc000360008)
        /go/pkg/mod/github.com/hashicorp/[email protected]/snapshot.go:190 +0x8c8
github.com/hashicorp/raft.(*Raft).runSnapshots(0xc000360008)
        /go/pkg/mod/github.com/hashicorp/[email protected]/snapshot.go:82 +0x3ee
github.com/hashicorp/raft.(*raftState).goFunc.func1()
        /go/pkg/mod/github.com/hashicorp/[email protected]/state.go:149 +0x56
created by github.com/hashicorp/raft.(*raftState).goFunc in goroutine 1
        /go/pkg/mod/github.com/hashicorp/[email protected]/state.go:147 +0x79

It seems to me that it's caused by gob/encoder still processing KV data for snapshot while it's being modified elsewhere (?)

Full dump attached.

crash.zip

[zajakala]

I experimented with using R lock for KV data (as I've seen in base.go) while encoder is processing data for snapshot in (*Fsm).Persist:

        var buffer bytes.Buffer
        f.RLock()
        defer f.RUnlock()
        err := gob.NewEncoder(&buffer).Encode(f.GetAll())

and it seems to help - but I have no idea if this is the correct approach or the right place to do it.