Besu looping with Failed to process block / World State Root does not match expected value

[Concurser]

For about the last 2 days my Besu installed started looping on block 21235203 and I am getting behind on the mainet. Initial sync took weeks to complete and I really would NOT like to have to repeat it all over again. I am writing a master thesis and had the idea to setup a local node because I need to query data a lot.

I tried backward sync but it got stuck in the same block number (21235203) with a "ERROR | AbstractBlockProcessor | failed persisting block" message.

I saw some similar issues dating back 2022, but they seemed fixes on Besu 23.

On besu I get this:

Nov 23 02:08:26 localhost.localdomain besu[90381]: 2024-11-23 02:08:26.339-03:00 | nioEventLoopGroup-3-7 | INFO | MainnetBlockValidator | Failed to process block 21235203 (0xff65b891caffff19ca82a2e46a61e5> Nov 23 02:08:26 localhost.localdomain besu[90381]: 2024-11-23 02:08:26.790-03:00 | nioEventLoopGroup-3-7 | ERROR | AbstractBlockProcessor | failed persisting block Nov 23 02:08:26 localhost.localdomain besu[90381]: java.lang.RuntimeException: World State Root does not match expected value, header 0x233a423c4765889625e47b3c360368658d3ba146a2f154b06c0218d087cb011b calc>

Here is my regular Teku output:
Nov 23 02:19:23 localhost.localdomain teku[90115]: 02:19:23.141 INFO - Syncing *** Target slot: 10459595, Head slot: 10459594, Waiting for execution layer sync, Connected peers: 74 Nov 23 02:19:35 localhost.localdomain teku[90115]: 02:19:35.149 INFO - Syncing *** Target slot: 10459596, Head slot: 10459595, Waiting for execution layer sync, Connected peers: 59 Nov 23 02:19:39 localhost.localdomain teku[90115]: 02:19:39.416 WARN - Late Block Import *** Block: d4402eb0232338df0f50fba606d70ea41d360f56962025e4fcb6d5d2dbba1837 (10459596) Proposer: 1324120 Result: success Timings: arrival 2326ms, gossip_validation +8ms, pre-state_retrieved +3ms, processed +170ms, execution_payload_result_received +0ms, data_availability_checked +1890ms, begin_importing +0ms, transaction_prepared +0ms, transaction_committed +0ms, completed +19ms Nov 23 02:19:47 localhost.localdomain teku[90115]: 02:19:47.139 INFO - Syncing *** Target slot: 10459597, Head slot: 10459596, Waiting for execution layer sync, Connected peers: 43 Nov 23 02:19:59 localhost.localdomain teku[90115]: 02:19:59.136 INFO - Syncing *** Target slot: 10459598, Head slot: 10459597, Waiting for execution layer sync, Connected peers: 40 Nov 23 02:20:11 localhost.localdomain teku[90115]: 02:20:11.140 INFO - Syncing *** Target slot: 10459599, Head slot: 10459598, Waiting for execution layer sync, Connected peers: 66

My Besu service file:
`[Unit]
Description=Hyperledger Besu Ethereum Client
After=network.target

[Service]
User=root
Environment="LD_PRELOAD=/usr/lib64/libjemalloc.so.2"
ExecStart=/usr/local/bin/besu/bin/besu --data-path=/besu-data --network=mainnet --sync-mode=FAST --metrics-enabled --metrics-host=0.0.0.0 --metrics-port=9545 --host-allowlist=* --rpc-http-enabled --rpc-http-port=8545 --rpc-http-api=ADMIN,NET,ETH,WEB3 --rpc-http-host=0.0.0.0 --engine-jwt-secret=/etc/teku/jwt.hex --max-peers=150

Restart=on-failure
LimitNOFILE=8192

[Install]
WantedBy=multi-user.target`

My system
Around 1.4TB were taken by besu data folder so far.
Red Hat Enterprise Linux release 9.4 (Plow)
Besu Version: besu/v24.9.1/linux-x86_64/openjdk-java-21
Teku version: teku/v24.8.0/linux-x86_64/-redhatinc-openjdk64bitservervm-java-21
CPU and RAM: 8 cores/32 GB of RAM
Some latency of 100ms on sda on write requests (got from iostat)
Besu data folder is on HDD storage (4 tb ST4000VX016-3CV104)

[siladu]

Hi @Concurser I would like to understand your use case as it will help us advise you on the appropriate configuration for Besu.

The main issue here appears to be the use of sync-mode=FAST. This is an old (soon to be deprecated) sync mode and has long since been superseded by sync-mode=SNAP, particularly for mainnet because FAST is (ironically) incredibly slow for mainnet.

If you change to use SNAP (or remove sync-mode since it is the default for mainnet) and resync by deleting your database, then you should be in sync within a couple of days, hardware dependent. The HDD may slow things down however, we normally recommend an SSD.
Check this page for fairly up to date requirements: https://ethdocker.com/Usage/ResourceUsage/#disk-ram-cpu-requirements
Notably disk should only take 1.2 TB, maybe usage of FAST is causing that to be slightly higher.

What data do you need to query? By default I expect you are using data-storage-format=BONSAI which will limit you to querying state only for the most recent 512 blocks. You can still query blockchain data from genesis, just not the state.

If you need to query historical state beyond the latest 512 blocks as well, then the appropriate configuration is an archive node, which would require FULL sync. However, that mode us currently undergoing some work and could take a couple of months to get in sync. Best to have a conversation with us on Hyperledger Discord first before embarking on the longer archive node option I think.

There is also sync-mode=CHECKPOINT which will be the fastest way to get in sync but you will be missing blockchain data from before the checkpoint (~October 2021).

More details here: https://besu.hyperledger.org/public-networks/concepts/node-sync

Since you have 32GB, you can probably add some more options to Besu to improve performance and sync even faster: https://besu.hyperledger.org/public-networks/how-to/troubleshoot/performance
in particular --Xplugin-rocksdb-high-spec-enabled and setting the JVM option -Xmx8g as per https://besu.hyperledger.org/public-networks/how-to/configure-java/manage-memory

Hope this helps.