Unable to create block because world state is not available

[junichi-tanaka]

Hi team, I'm facing the following error. Could someone give me any advice to resolve the issue?

Background:

• I'm running 4 besu nodes with private network on Azure Kubernetes Service.
• After a node maintenance reboot, all besu pods were recreated and cause the same error from all pods.
• I've tried to restart all pods but I'm still facing the same error.

Setting logging level to INFO
2024-11-27 05:58:18.305+00:00 | main | INFO  | Besu | Starting Besu
2024-11-27 05:58:18.877+00:00 | main | WARN  | Besu | --rpc-ws-host, --rpc-ws-port, --rpc-ws-api and --rpc-ws-authentication-enabled has been ignored because --rpc-ws-enabled was not defined on the command line.
2024-11-27 05:58:18.882+00:00 | main | WARN  | Besu | --graphql-http-host, --graphql-http-port and --graphql-http-cors-origins has been ignored because --graphql-http-enabled was not defined on the command line.
2024-11-27 05:58:19.203+00:00 | main | WARN  | Besu | Forcing --bonsai-limit-trie-logs-enabled=false, since it cannot be enabled with --sync-mode=FULL and --data-storage-format=BONSAI.
2024-11-27 05:58:19.203+00:00 | main | WARN  | Besu | Permissions are disabled. Cannot enable PERM APIs when not using Permissions.
2024-11-27 05:58:19.205+00:00 | main | INFO  | Besu | Connecting to 0 static nodes.
2024-11-27 05:58:19.209+00:00 | main | WARN  | Besu | Forcing price bump for transaction replacement to 0, since min-gas-price is set to 0
2024-11-27 05:58:19.210+00:00 | main | WARN  | Besu | Forcing tx-pool-min-gas-price=0, since it cannot be greater than the value of min-gas-price
2024-11-27 05:58:19.369+00:00 | main | INFO  | Besu |
####################################################################################################
#                                                                                                  #
# Besu version 24.9.1                                                                              #
#                                                                                                  #
# Configuration:                                                                                   #
# Network: Custom genesis file                                                                     #
# /etc/besu/genesis.json                                                                           #
# Network Id: 2028                                                                                 #
# Data storage: Bonsai                                                                             #
# Sync mode: Full                                                                                  #
# RPC HTTP APIs: DEBUG,ETH,ADMIN,WEB3,IBFT,NET,TRACE,EEA,PRIV,QBFT,PERM,TXPOOL                     #
# RPC HTTP port: 8545                                                                              #
# Using LAYERED transaction pool implementation                                                    #
# Using STACKED worldstate update mode                                                             #
#                                                                                                  #
# Host:                                                                                            #
# Java: openjdk-java-21                                                                            #
# Maximum heap size: 0.45 GB                                                                       #
# OS: linux-x86_64                                                                                 #
# glibc: 2.39                                                                                      #
# jemalloc: 5.3.0-0-g54eaed1d8b56b1aa528be3bdd1877e59c56fa90c                                      #
# Total memory: 15.62 GB                                                                           #
# CPU cores: 4                                                                                     #
#                                                                                                  #
# Plugin Registration Summary:                                                                     #
# No plugins have been registered.                                                                 #
# TOTAL = 0 of 0 plugins successfully registered.                                                  #
#                                                                                                  #
####################################################################################################
...
2024-11-27 05:58:22.380+00:00 | main | INFO  | DefaultSynchronizer | Starting synchronizer.
2024-11-27 05:58:22.383+00:00 | main | INFO  | TransactionPoolFactory | Enabling transaction handling following initial sync
2024-11-27 05:58:22.383+00:00 | main | INFO  | QbftBesuControllerBuilder | Starting QBFT mining coordinator following initial sync
2024-11-27 05:58:38.094+00:00 | main | INFO  | FullSyncDownloader | Starting full sync.
2024-11-27 05:58:38.098+00:00 | main | INFO  | Runner | Ethereum main loop is up.
2024-11-27 05:58:38.106+00:00 | BftProcessorExecutor-QBFT-0 | INFO  | AbstractBlockCreator | Unable to create block because world state is not available
2024-11-27 05:58:38.107+00:00 | BftProcessorExecutor-QBFT-0 | ERROR | EventMultiplexer | State machine threw exception while processing event \{BlockTimerExpiry{Round Identifier=ConsensusRoundIdentifier{Sequence=3684383, Round=0}}\}
java.util.concurrent.CancellationException: World state not available for block 3684382 with state root 0xdec05459e22919757f751b0dccc2e77a2d15bfbe0b4fe1eb6e2cfa819de13da3
        at org.hyperledger.besu.ethereum.blockcreation.AbstractBlockCreator.lambda$duplicateWorldStateAtParent$2(AbstractBlockCreator.java:415)
        at java.base/java.util.Optional.orElseThrow(Optional.java:403)
        at org.hyperledger.besu.ethereum.blockcreation.AbstractBlockCreator.duplicateWorldStateAtParent(AbstractBlockCreator.java:410)
        at org.hyperledger.besu.ethereum.blockcreation.AbstractBlockCreator.createBlock(AbstractBlockCreator.java:195)
        at org.hyperledger.besu.ethereum.blockcreation.AbstractBlockCreator.createBlock(AbstractBlockCreator.java:149)
        at org.hyperledger.besu.consensus.common.bft.blockcreation.BftBlockCreator.createBlock(BftBlockCreator.java:89)
        at org.hyperledger.besu.consensus.qbft.statemachine.QbftRound.createAndSendProposalMessage(QbftRound.java:142)
        at org.hyperledger.besu.consensus.qbft.statemachine.QbftBlockHeightManager.handleBlockTimerExpiry(QbftBlockHeightManager.java:140)
        at org.hyperledger.besu.consensus.common.bft.statemachine.BaseBftController.handleBlockTimerExpiry(BaseBftController.java:167)
        at org.hyperledger.besu.consensus.common.bft.EventMultiplexer.handleBftEvent(EventMultiplexer.java:65)
        at java.base/java.util.Optional.ifPresent(Optional.java:178)
        at org.hyperledger.besu.consensus.common.bft.BftProcessor.run(BftProcessor.java:68)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
        at java.base/java.lang.Thread.run(Thread.java:1583)

Versions (Add all that apply)

• Software version: besu/v24.9.1/linux-x86_64/openjdk-java-21 (hyperledger/besu:24.9.1 docker image)
• Java version: openjdk version "21.0.4" 2024-07-16
• OS Name & Version: Ubuntu 24.04 LTS
• Kernel Version: Linux besu-validator-1-0 5.15.0-1074-azure New Maintainer candidate: Ivaylo Kirilov #83-Ubuntu SMP Wed Oct 2 18:14:49 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Additional Information (Add any of the following or anything else that may be relevant)

• Besu setup info - genesis file, config options

genesis file

{
  "config": {
    "chainId": 2028,
    "muirglacierblock": 0,
    "qbft": {
      "blockperiodseconds": 1,
      "epochlength": 30000,
      "requesttimeoutseconds": 4
    }
  },
  "nonce": "0x0",
  "timestamp": "0x58ee40ba",
  "gasLimit": "0x1fffffffffffff",
  "difficulty": "0x1",
  "mixHash": "0x63746963616c2062797a616e74696e65206661756c7420746f6c6572616e6365",
  "coinbase": "0x0000000000000000000000000000000000000000",
  "extraData": "0xf87aa00000000000000000000000000000000000000000000000000000000000000000f85494dd5e5d3808917ab4224872e1320c3f79c26dd43394f2418a1c31108f844feb59c1a269313d2c3aced494fe60c404d7b949efb5ff12b9048a9bfa60ec136094de37c8946d9ff06ba083ed9ea314052f0cfb16a4c080c0",
  "alloc": {
  ...
  }
}

config file

# Every possible CLI should be in this file.
#
# Please use a plausible value, besu has to at least be able to parse it.
# If it is a multi-valued CLI make it a TOML array.
# If it is a number or boolean make it a number or boolean
# All other config options are strings, and must be quoted.

# Node Information
data-path="/data"
genesis-file="/etc/besu/genesis.json"

# TODO : insert as command line & mount Volume
# rpc-http-authentication-credentials-file="none"
# TODO : insert as command line & mount Volume
# rpc-ws-authentication-credentials-file="none"
logging="INFO"

min-gas-price=0
# Transaction Pool
#tx-pool-retention-hours=999
#tx-pool-max-size=1024

# P2P network
p2p-enabled=true
discovery-enabled=true
#p2p-host="0.0.0.0"
p2p-port=30303
max-peers=25
host-allowlist=["*"]

# JSON-RPC
rpc-http-enabled=true
rpc-http-host="0.0.0.0"
rpc-http-port=8545
rpc-http-api=["DEBUG", "ETH", "ADMIN", "WEB3", "IBFT", "NET", "TRACE", "EEA", "PRIV", "QBFT", "PERM", "TXPOOL"]
rpc-http-cors-origins=["*"]
rpc-http-authentication-enabled=false

# GRAPHQL-RPC
graphql-http-enabled=false
graphql-http-host="0.0.0.0"
graphql-http-port=8547
graphql-http-cors-origins=["*"]

# WebSockets API
rpc-ws-enabled=false
rpc-ws-host="0.0.0.0"
rpc-ws-port=8546
rpc-ws-api=["DEBUG","ETH", "ADMIN", "WEB3", "IBFT", "NET", "TRACE", "EEA", "PRIV", "QBFT", "PERM", "TXPOOL"]
rpc-ws-authentication-enabled=false

# Metrics
metrics-enabled=true
metrics-host="0.0.0.0"
metrics-port=9545

# Permissioning
#permissions-nodes-config-file-enabled=true
#permissions-nodes-config-file="/etc/permissions/nodes-allowlist.yml"

Thanks in advance.