[Oracle NST] Consensus Failure after Withdraw NST

[cloud8little]

Summary of Bug

Deposit multiple NST record in the exocore system, then withdraw one of the record, wait for several blocks, consensus failure.

Version

https://github.com/leonz789/exocore/releases/tag/v1.0.10
price feeder: ExocoreNetwork/price-feeder@b3670da

Steps to Reproduce

Steps

1. Start four validators network using https://github.com/cloud8little/ExocoreValidators,disable --exocore option. exocore binary: https://github.com/leonz789/exocore/releases/tag/v1.0.10

2. Replace all genesis file with genesis_v7_delegationfix.json

3. Start price feeder for four nodes separately.
   price feeder binary: ExocoreNetwork/price-feeder@b3670da

4. deposit nst for staker 1.

Deposit 32ETH NST with staker 1, the validator is running.
assetcli depositNST --rpcUrl http://localhost:9545 --staker 0xa53f68563D22EB0dAFAA871b6C08a6852f91d627 --amount 32000000000000000000 --privateKey C26A874A75B028638D477DDF31EB8627899CB505798DF70D2DD2A631F9CAE7A4  --pubkey 00000000000000000000000000000000000000000000000000000000001CDD66 --layerZeroID 40217

5. deposit nst for staker 2.

Deposit 32ETH NST with staker 2, the validiator is slashed, effective balance is 31ETH.
assetcli depositNST --rpcUrl http://localhost:9545 --staker 0x0B34c4D876cd569129CF56baFAbb3F9E97A4fF42 --amount 32000000000000000000 --privateKey C26A874A75B028638D477DDF31EB8627899CB505798DF70D2DD2A631F9CAE7A4  --pubkey 000000000000000000000000000000000000000000000000000000000019FB60 --layerZeroID 40217

6. Wait for about 7 minutes, the staker assets Info of staker 2 should be updated as 31ETH.

exocored q assets QueStakerAssetInfos 0x0B34c4D876cd569129CF56baFAbb3F9E97A4fF42_0x9d19 --node http://localhost:20000
asset_infos:
- asset_id: 0xeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee_0x9d19
  info:
    pending_undelegation_amount: "0"
    total_deposit_amount: "31000000000000000000"
    withdrawable_amount: "31000000000000000000"

7. withdrawNST for staker1. height: 166

assetcli withdrawNST --rpcUrl http://localhost:9545 --staker 0xa53f68563D22EB0dAFAA871b6C08a6852f91d627 --amount 32000000000000000000 --privateKey C26A874A75B028638D477DDF31EB8627899CB505798DF70D2DD2A631F9CAE7A4 --pubkey 000000000000000000000000000000000000000000000000000000000019FB60 --layerZeroID 40217

8. Wait for around 2 blocks, consensus failure for all nodes. height: 171

9:42AM INF commit round with price from previous baseBlock=165 feederID=1 heigth=171 module=x/oracle roundID=16
9:42AM INF commit round with price from previous baseBlock=165 feederID=3 heigth=171 module=x/oracle roundID=6
9:42AM ERR CONSENSUS FAILURE!!! err="runtime error: index out of range [1] with length 1" module=consensus server=node stack="goroutine 140 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x5e\ngithub.com/cometbft/cometbft/consensus.(*State).receiveRoutine.func2()\n\t/root/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:736 +0x46\npanic({0x31c4880?, 0xc007270cd8?})\n\t/usr/local/go/src/runtime/panic.go:770 +0x132\ngithub.com/ExocoreNetwork/exocore/x/oracle/keeper.parseBalanceChangeCapped({0xc00563da70, 0x21, 0x21}, {{0xc006321800?, 0x4482160?, 0xc000872810?}})\n\t/go/src/github.com/ExocoreNetwork/exocore/x/oracle/keeper/native_token.go:409 +0x356\ngithub.com/ExocoreNetwork/exocore/x/oracle/keeper.Keeper.UpdateNSTByBalanceChange({{0x44c7510, 0xc00169fe00}, {0x4482070, 0xc0008727a0}, {0x4482160, 0xc000872810}, {{0x44c7510, 0xc00169fe00}, 0xc001612560, {0x4482070, ...}, ...}, ...}, ...)\n\t/go/src/github.com/ExocoreNetwork/exocore/x/oracle/keeper/native_token.go:280 +0x14a\ngithub.com/ExocoreNetwork/exocore/x/oracle/keeper.Keeper.AppendPriceTR({{0x44c7510, 0xc00169fe00}, {0x4482070, 0xc0008727a0}, {0x4482160, 0xc000872810}, {{0x44c7510, 0xc00169fe00}, 0xc001612560, {0x4482070, ...}, ...}, ...}, ...)\n\t/go/src/github.com/ExocoreNetwork/exocore/x/oracle/keeper/prices.go:217 +0x61a\ngithub.com/ExocoreNetwork/exocore/x/oracle/keeper.Keeper.GrowRoundID({{0x44c7510, 0xc00169fe00}, {0x4482070, 0xc0008727a0}, {0x4482160, 0xc000872810}, {{0x44c7510, 0xc00169fe00}, 0xc001612560, {0x4482070, ...}, ...}, ...}, ...)\n\t/go/src/github.com/ExocoreNetwork/exocore/x/oracle/keeper/prices.go:233 +0x188\ngithub.com/ExocoreNetwork/exocore/x/oracle/keeper/feedermanagement.(*FeederManager).commitRounds(_, {{0x44aa170, 0x64bf060}, {0x44c7cf0, 0xc005c05880}, {{0xb, 0x0}, {0xc007dc8b58, 0x14}, 0xab, ...}, ...})\n\t/go/src/github.com/ExocoreNetwork/exocore/x/oracle/keeper/feedermanagement/feedermanager.go:255 +0x976\ngithub.com/ExocoreNetwork/exocore/x/oracle/keeper/feedermanagement.(*FeederManager).updateAndCommitRounds(_, {{0x44aa170, 0x64bf060}, {0x44c7cf0, 0xc005c05880}, {{0xb, 0x0}, {0xc007dc8b58, 0x14}, 0xab, ...}, ...})\n\t/go/src/github.com/ExocoreNetwork/exocore/x/oracle/keeper/feedermanagement/feedermanager.go:534 +0x85\ngithub.com/ExocoreNetwork/exocore/x/oracle/keeper/feedermanagement.(*FeederManager).EndBlock(_, {{0x44aa170, 0x64bf060}, {0x44c7cf0, 0xc005c05880}, {{0xb, 0x0}, {0xc007dc8b58, 0x14}, 0xab, ...}, ...})\n\t/go/src/github.com/ExocoreNetwork/exocore/x/oracle/keeper/feedermanagement/feedermanager.go:79 +0xc5\ngithub.com/ExocoreNetwork/exocore/x/oracle.AppModule.EndBlock(...)\n\t/go/src/github.com/ExocoreNetwork/exocore/x/oracle/module.go:150\ngithub.com/cosmos/cosmos-sdk/types/module.(*Manager).EndBlock(_, {{0x44aa170, 0x64bf060}, {0x44c7cf0, 0xc005c05880}, {{0xb, 0x0}, {0xc007dc8b58, 0x14}, 0xab, ...}, ...}, ...)\n\t/root/go/pkg/mod/github.com/evmos/[email protected]/types/module/module.go:583 +0x1bf\ngithub.com/ExocoreNetwork/exocore/app.(*ExocoreApp).EndBlocker(...)\n\t/go/src/github.com/ExocoreNetwork/exocore/app/app.go:1162\ngithub.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).EndBlock(0xc00166c780, {0x55?})\n\t/root/go/pkg/mod/github.com/evmos/[email protected]/baseapp/abci.go:219 +0x125\ngithub.com/cometbft/cometbft/abci/client.(*localClient).EndBlockSync(0xc002a801e0, {0x0?})\n\t/root/go/pkg/mod/github.com/cometbft/[email protected]/abci/client/local_client.go:291 +0xd9\ngithub.com/cometbft/cometbft/proxy.(*appConnConsensus).EndBlockSync(0xc002a7a078, {0x20?})\n\t/root/go/pkg/mod/github.com/cometbft/[email protected]/proxy/app_conn.go:111 +0x14c\ngithub.com/cometbft/cometbft/state.execBlockOnProxyApp({0x44aa1a8, 0xc004dd7e20}, {0x44c3ba0, 0xc002a7a078}, 0xc00166da40, {0x44c8910, 0xc004df3a40}, 0x1)\n\t/root/go/pkg/mod/github.com/cometbft/[email protected]/state/execution.go:383 +0x6b0\ngithub.com/cometbft/cometbft/state.(*BlockExecutor).ApplyBlock(_, {{{0xb, 0x0}, {0xc004de5674, 0x6}}, {0xc004db5260, 0x14}, 0x1, 0xaa, {{0xc008441460, ...}, ...}, ...}, ...)\n\t/root/go/pkg/mod/github.com/cometbft/[email protected]/state/execution.go:197 +0x18e\ngithub.com/cometbft/cometbft/consensus.(*State).finalizeCommit(0xc00084ca88, 0xab)\n\t/root/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:1710 +0xa4c\ngithub.com/cometbft/cometbft/consensus.(*State).tryFinalizeCommit(0xc00084ca88, 0xab)\n\t/root/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:1619 +0x2e8\ngithub.com/cometbft/cometbft/consensus.(*State).enterCommit.func1()\n\t/root/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:1554 +0x9c\ngithub.com/cometbft/cometbft/consensus.(*State).enterCommit(0xc00084ca88, 0xab, 0x0)\n\t/root/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:1592 +0xc3b\ngithub.com/cometbft/cometbft/consensus.(*State).addVote(0xc00084ca88, 0xc00182a6e0, {0xc0067b8570, 0x28})\n\t/root/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:2223 +0x182d\ngithub.com/cometbft/cometbft/consensus.(*State).tryAddVote(0xc00084ca88, 0xc00182a6e0, {0xc0067b8570?, 0x0?})\n\t/root/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:2012 +0x26\ngithub.com/cometbft/cometbft/consensus.(*State).handleMsg(0xc00084ca88, {{0x44778e0, 0xc001d89918}, {0xc0067b8570, 0x28}})\n\t/root/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:865 +0x39c\ngithub.com/cometbft/cometbft/consensus.(*State).receiveRoutine(0xc00084ca88, 0x0)\n\t/root/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:772 +0x3f1\ncreated by github.com/cometbft/cometbft/consensus.(*State).OnStart in goroutine 108\n\t/root/go/pkg/mod/github.com/cometbft/[email protected]/consensus/state.go:383 +0x10c\n"
9:42AM INF service stop impl=baseWAL module=consensus msg="Stopping baseWAL service" server=node wal=node2/data/cs.wal/wal
9:42AM INF service stop impl=Group module=consensus msg="Stopping Group service" server=node wal=node2/data/cs.wal/wal
9:42AM INF Timed out dur=3000 height=171 module=consensus round=0 server=node step=RoundStepPropose

Logs attachment:
logs_20250114.zip

Screenshots

Additional context

---

For Admin Use

• Not duplicate issue
• Appropriate labels applied
• Appropriate contributors tagged
• Contributor assigned/self-assigned

[leonz789]

fixed by latest commit from #275 and price-feeder