[Bug]: Panic - "err": "There's less MTM wins than losses, even accounting for decimals"

[Sohill-Patel]

Problem encountered

panic in system-test nightly run

Observed behaviour

2023-11-25T02:38:12.196Z ERROR tendermint consensus/state.go:800 CONSENSUS FAILURE!!! {"err": "There's less MTM wins than losses, even accounting for decimals", "stack": "goroutine 250 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x65\ngithub.com/cometbft/cometbft/consensus.(*State).receiveRoutine.func2()\n\t/jenkins/GOPATH/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:800 +0x4c\npanic({0x4c6c600, 0xc004a69a00})\n\t/usr/local/go/src/runtime/panic.go:884 +0x213\ngo.uber.org/zap/zapcore.CheckWriteAction.OnWrite(0x2?, 0x2?, {0x0?, 0x0?, 0xc003ba2e00?})\n\t/jenkins/GOPATH/pkg/mod/go.uber.org/[email protected]/zapcore/entry.go:198 +0x65\ngo.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc003445860, {0xc00548cd00, 0x2, 0x2})\n\t/jenkins/GOPATH/pkg/mod/go.uber.org/[email protected]/zapcore/entry.go:264 +0x3ec\ngo.uber.org/zap.(*Logger).Panic(0xc003cf2760?, {0x5734ab6?, 0xc003cf28a0?}, {0xc00548cd00, 0x2, 0x2})\n\t/jenkins/GOPATH/pkg/mod/go.uber.org/[email protected]/logger.go:258 +0x59\ncode.vegaprotocol.io/vega/core/settlement.(*Engine).SettleMTM(0xc004241a00, {0x5df99f0, 0xc00470fbf0}, 0xc0057889a0, {0xc004d7be00, 0x6, 0x7f881c0afb78?})\n\t/jenkins/workspace/common/system-tests-wrapper/vega/core/settlement/engine.go:387 +0x18aa\ncode.vegaprotocol.io/vega/core/execution/future.(*Market).confirmMTM(0xc0028ba580, {0x5df99f0, 0xc00470fbf0}, 0x0)\n\t/jenkins/workspace/common/system-tests-wrapper/vega/core/execution/future/market.go:2290 +0x207\ncode.vegaprotocol.io/vega/core/execution/future.(*Market).BlockEnd(0xc0028ba580, {0x5df99f0?, 0xc00470fbf0?})\n\t/jenkins/workspace/common/system-tests-wrapper/vega/core/execution/future/market.go:968 +0x48e\ncode.vegaprotocol.io/vega/core/execution.(*Engine).BlockEnd(0xc000143080?, {0x5df99f0, 0xc00470fbf0})\n\t/jenkins/workspace/common/system-tests-wrapper/vega/core/execution/engine.go:1479 +0x46\ncode.vegaprotocol.io/vega/core/processor.(*App).OnEndBlock(0xc000276700, 0x7f881c0aa4f8?)\n\t/jenkins/workspace/common/system-tests-wrapper/vega/core/processor/abci.go:924 +0x7c2\ncode.vegaprotocol.io/vega/core/blockchain/abci.(*App).FinalizeBlock(0xc000f1bc80, {0x7f8848411d28?, 0x30?}, 0xc00127b2c0)\n\t/jenkins/workspace/common/system-tests-wrapper/vega/core/blockchain/abci/abci.go:183 +0x966\ncode.vegaprotocol.io/vega/cmd/vega/commands/node.(*appW).FinalizeBlock(0xc00158c548?, {0x5df9980?, 0xc00007e148?}, 0x5075d80?)\n\t/jenkins/workspace/common/system-tests-wrapper/vega/cmd/vega/commands/node/app_wrapper.go:91 +0x2b\ngithub.com/cometbft/cometbft/abci/client.(*localClient).FinalizeBlock(0x5de6430?, {0x5df9980?, 0xc00007e148?}, 0xc00127b2c0?)\n\t/jenkins/GOPATH/pkg/mod/github.com/cometbft/[email protected]/abci/client/local_client.go:185 +0xdb\ngithub.com/cometbft/cometbft/proxy.(*appConnConsensus).FinalizeBlock(0xc0004a36b0, {0x5df9980, 0xc00007e148}, 0x6?)\n\t/jenkins/GOPATH/pkg/mod/github.com/cometbft/[email protected]/proxy/app_conn.go:104 +0x102\ngithub.com/cometbft/cometbft/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x1}, {0xc000d46eb0, 0x6}}, {0xc000d46ec0, 0xb}, 0x1, 0x240f, {{0xc004b85c80, ...}, ...}, ...}, ...)\n\t/jenkins/GOPATH/pkg/mod/github.com/cometbft/[email protected]/state/execution.go:215 +0x686\ngithub.com/cometbft/cometbft/consensus.(*State).finalizeCommit(0xc00056ea80, 0x2410)\n\t/jenkins/GOPATH/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:1770 +0xb9d\ngithub.com/cometbft/cometbft/consensus.(*State).tryFinalizeCommit(0xc00056ea80, 0x2410)\n\t/jenkins/GOPATH/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:1681 +0x2ff\ngithub.com/cometbft/cometbft/consensus.(*State).enterCommit.func1()\n\t/jenkins/GOPATH/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:1616 +0xaa\ngithub.com/cometbft/cometbft/consensus.(*State).enterCommit(0xc00056ea80, 0x2410, 0x1)\n\t/jenkins/GOPATH/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:1654 +0xccf\ngithub.com/cometbft/cometbft/consensus.(*State).addVote(0xc00056ea80, 0xc0040f7450, {0xc0005d20f0, 0x28})\n\t/jenkins/GOPATH/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:2323 +0x1cff\ngithub.com/cometbft/cometbft/consensus.(*State).tryAddVote(0xc00056ea80, 0xc0040f7450, {0xc0005d20f0?, 0xc00158dc08?})\n\t/jenkins/GOPATH/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:2055 +0x2c\ngithub.com/cometbft/cometbft/consensus.(*State).handleMsg(0xc00056ea80, {{0x5dd0f20, 0xc005b2a048}, {0xc0005d20f0, 0x28}})\n\t/jenkins/GOPATH/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:928 +0x3ff\ngithub.com/cometbft/cometbft/consensus.(*State).receiveRoutine(0xc00056ea80, 0x0)\n\t/jenkins/GOPATH/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:835 +0x3f0\ncreated by github.com/cometbft/cometbft/consensus.(*State).OnStart\n\t/jenkins/GOPATH/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:397 +0x12d\n"}

Expected behaviour

no panic

Steps to reproduce

1. system-test nightly run on Saturday reported the issue but did not occur on Sunday/Monday so not sure of the cause

Software version

dev branch

Failing test

No response

Jenkins run

https://jenkins.vega.rocks/job/common/job/system-tests-nightly/111/

Configuration used

No response

Relevant log output

No response

[EVODelavega]

Related to #10168