Syncing issue from snapshot

[JohnsonCaii]

Hi I'm syncing theta from snapshot following the guide.

I've tried multiple times, it said Block is invalid block.Hash=0x5492791543ea6f6b490fbf400a437363287f3b9a5b027e822e34e98ed7f31ad1 every time when processed to 11604501 block height.

And the log message:

[2021-08-18 03:37:12]  INFO [consensus] Finalizing block block.Hash=0xa675bf8806ab526c976153fd89902bbce0785fe0a8af410385e56790b49ff3b1 block.Height=11604498
[2021-08-18 03:37:12]  INFO [consensus] Notified finalized block, height=11604498
[2021-08-18 03:37:12]  INFO [rpc] Processing finalized block, height=11604498
[2021-08-18 03:37:12]  INFO [rpc] Done processing finalized block, height=11604498
[2021-08-18 03:37:12]  INFO [request] Sync progress current chain tip=0x9d21023f02175a4114f503182c11cdf5188b7e7d537fe9b9c9b631ec9f06f3ed fast sync tip=11604500 pending block hashes=21
[2021-08-18 03:37:14]  WARN [consensus] Failed to load guardian pool block.GuardianVotes=AggregatedVotes{Block: 0x074bc5b91a93560c590faa2ca3a784eade7bf1a4d357ada17b6c5ef41673917f, Gcp: 0x7375c445207f77eaa790ce5d446fd9c6744beef63d2f4f671abd8236d8429679,  Multiplies: [...alot of binary numbers...]} block.Hash=0x5492791543ea6f6b490fbf400a437363287f3b9a5b027e822e34e98ed7f31ad1 block.Height=11604501 error=Result{code:100000, message:signature verification failed}
[2021-08-18 03:37:14]  WARN [consensus] Block is invalid block.Hash=0x5492791543ea6f6b490fbf400a437363287f3b9a5b027e822e34e98ed7f31ad1
[2021-08-18 03:37:19]  INFO [request] Sync progress current chain tip=0xcf0a340ee615b8314b755c534b49e3aea327ec908727917298be667a2df25f25 fast sync tip=11604501 pending block hashes=22

Then the process kept printing these messages:

[2021-08-18 03:43:22]  INFO [request] Sync progress current chain tip=0xcf0a340ee615b8314b755c534b49e3aea327ec908727917298be667a2df25f25 fast sync tip=11604501 pending block hashes=269
[2021-08-18 03:43:24]  INFO [p2p] Already has sufficient number of peers, numPeers: 24, sufficientNumPeers: 24
[2021-08-18 03:43:29]  INFO [request] Sync progress current chain tip=0xcf0a340ee615b8314b755c534b49e3aea327ec908727917298be667a2df25f25 fast sync tip=11604501 pending block hashes=275
[2021-08-18 03:43:36]  INFO [request] Sync progress current chain tip=0xcf0a340ee615b8314b755c534b49e3aea327ec908727917298be667a2df25f25 fast sync tip=11604501 pending block hashes=283
[2021-08-18 03:43:43]  INFO [request] Sync progress current chain tip=0xcf0a340ee615b8314b755c534b49e3aea327ec908727917298be667a2df25f25 fast sync tip=11604501 pending block hashes=288
[2021-08-18 03:43:50]  INFO [request] Sync progress current chain tip=0xcf0a340ee615b8314b755c534b49e3aea327ec908727917298be667a2df25f25 fast sync tip=11604501 pending block hashes=295
[2021-08-18 03:43:54]  INFO [p2p] Already has sufficient number of peers, numPeers: 24, sufficientNumPeers: 24
[2021-08-18 03:43:57]  INFO [request] Sync progress current chain tip=0xcf0a340ee615b8314b755c534b49e3aea327ec908727917298be667a2df25f25 fast sync tip=11604501 pending block hashes=301
[2021-08-18 03:43:59]  WARN [p2p] sendRoutine error: write tcp 172.18.0.135:34108->52.14.127.89:21000: write: broken pipe
[2021-08-18 03:43:59]  WARN [p2p] Connection error: write tcp 172.18.0.135:34108->52.14.127.89:21000: write: broken pipe
[2021-08-18 03:43:59]  WARN [p2p] Stopping connection, local: 172.18.0.135:34108, remote: 52.14.127.89:21000
[2021-08-18 03:43:59]  INFO [p2p] HandlePeerWithErrors, peerRemoteAddress: 52.14.127.89:21000
[2021-08-18 03:43:59]  INFO [p2p] HandlePeerWithErrors, lookedUpPeerRemoteAddress: 52.14.127.89:21000
[2021-08-18 03:43:59]  INFO [p2p] Deleted peer 0x2495afB3d3D105C04b51Dc1C11ACd27F0751B2Bf from the peer table
[2021-08-18 03:43:59]  WARN [p2p] Stopping connection, local: 172.18.0.135:34108, remote: 52.14.127.89:21000
[2021-08-18 03:43:59]  WARN [p2p] Failed to close connection: close tcp 172.18.0.135:34108->52.14.127.89:21000: use of closed network connection
[2021-08-18 03:43:59]  INFO [p2p] Lost connection to peer 0x2495afB3d3D105C04b51Dc1C11ACd27F0751B2Bf with IP address 52.14.127.89:21000, trying to re-connect
[2021-08-18 03:43:59]  INFO [p2p] Handshaking with 52.14.127.89:21000...
[2021-08-18 03:43:59]  INFO [p2p] Peer ChainID: mainnet
[2021-08-18 03:43:59]  INFO [p2p] Peer Type: 1
[2021-08-18 03:43:59]  INFO [p2p] Using encrypted transport for peer: 0x2495afB3d3D105C04b51Dc1C11ACd27F0751B2Bf
[2021-08-18 03:43:59]  INFO [p2p] Handshake completed, target address: 52.14.127.89:21000, target public key: 044e3b8f98c1e86256d3b621eb3a2839b1e4faea0cbf41eb5305601c07ffd89b33acfc018c0828824cbd4270c19c578c156177a98c292829cbad52e7e41bb2f3ca, address: 0x2495afB3d3D105C04b51Dc1C11ACd27F0751B2Bf
[2021-08-18 03:43:59]  INFO [p2p] Successfully re-connected to peer 52.14.127.89:21000
[2021-08-18 03:44:00]  WARN [p2p] recvRoutine: failed to decode packet: <nil>, error: EOF
[2021-08-18 03:44:04]  INFO [request] Sync progress current chain tip=0xcf0a340ee615b8314b755c534b49e3aea327ec908727917298be667a2df25f25 fast sync tip=11604501 pending block hashes=306
[2021-08-18 03:44:11]  INFO [request] Sync progress current chain tip=0xcf0a340ee615b8314b755c534b49e3aea327ec908727917298be667a2df25f25 fast sync tip=11604501 pending block hashes=314
[2021-08-18 03:44:18]  INFO [request] Sync progress current chain tip=0xcf0a340ee615b8314b755c534b49e3aea327ec908727917298be667a2df25f25 fast sync tip=11604501 pending block hashes=320

The theta version:

$ theta version
[2021-08-18 03:45:39]  INFO Log settings: map[*:debug], *:debug
Version 3.1.0 dadc576ed7cc6a20f6ed4c174a3127e02737b933
Built at Mon Aug 16 06:28:40 UTC 202

What should I do to fix the stuck problem? please help!

[jieyilong]

Could you delete the node database folder (mainnet/walletnode/db) and the snapshot (mainnet/walletnode/snapshot). Then download the latest snapshot, and restart the node?

wget -O ../mainnet/walletnode/snapshot `curl -k https://mainnet-data.thetatoken.org/snapshot`

[JohnsonCaii]

Could you delete the node database folder (mainnet/walletnode/db) and the snapshot (mainnet/walletnode/snapshot). Then download the latest snapshot, and restart the node?

wget -O ../mainnet/walletnode/snapshot `curl -k https://mainnet-data.thetatoken.org/snapshot`

Thanks. I just redownloaded the snapshot and restart it, still the same.

But the invalid block reported has changed.

[2021-08-18 05:43:55]  WARN [consensus] Block is invalid block.Hash=0xafdcd83ad98fbd42f3fbccd013b12cade4b6985e0633d59c6109ea22c3aa5bc9

[2021-08-18 05:47:17]  WARN [consensus] Ignoring invalid vote err=Result{code:100000, message:Signature verification failed}
[2021-08-18 05:47:17]  WARN [consensus] Ignoring invalid vote err=Result{code:100000, message:Signature verification failed}
[2021-08-18 05:47:17]  WARN [consensus] Ignoring invalid vote err=Result{code:100000, message:Signature verification failed}
[2021-08-18 05:47:18]  WARN [consensus] Ignoring invalid vote err=Result{code:100000, message:Signature verification failed}
[2021-08-18 05:47:19]  WARN [consensus] Ignoring invalid vote err=Result{code:100000, message:Signature verification failed}
[2021-08-18 05:47:20]  WARN [consensus] Ignoring invalid vote err=Result{code:100000, message:Signature verification failed}
[2021-08-18 05:47:21]  INFO [request] Sync progress current chain tip=0x772ece8d90f1c5be984fea675cc33c6af4745cae59f32ea81d545d43f8a9b717 fast sync tip=11613801 pending block hashes=724
[2021-08-18 05:47:25]  WARN [consensus] Ignoring invalid vote err=Result{code:100000, message:Signature verification failed}
[2021-08-18 05:47:28]  WARN [consensus] Ignoring invalid vote err=Result{code:100000, message:Signature verification failed}
[2021-08-18 05:47:28]  INFO [request] Sync progress current chain tip=0x772ece8d90f1c5be984fea675cc33c6af4745cae59f32ea81d545d43f8a9b717 fast sync tip=11613801 pending block hashes=745
[2021-08-18 05:47:29]  WARN [consensus] Ignoring invalid vote err=Result{code:100000, message:Signature verification failed}
[2021-08-18 05:47:35]  INFO [request] Sync progress current chain tip=0x772ece8d90f1c5be984fea675cc33c6af4745cae59f32ea81d545d43f8a9b717 fast sync tip=11613801 pending block hashes=782
[2021-08-18 05:47:40]  INFO [p2p] Already has sufficient number of peers, numPeers: 25, sufficientNumPeers: 24
[2021-08-18 05:47:42]  INFO [request] Sync progress current chain tip=0x772ece8d90f1c5be984fea675cc33c6af4745cae59f32ea81d545d43f8a9b717 fast sync tip=11613801 pending block hashes=801
[2021-08-18 05:47:49]  INFO [request] Sync progress current chain tip=0x772ece8d90f1c5be984fea675cc33c6af4745cae59f32ea81d545d43f8a9b717 fast sync tip=11613801 pending block hashes=821
[2021-08-18 05:47:56]  INFO [request] Sync progress current chain tip=0x772ece8d90f1c5be984fea675cc33c6af4745cae59f32ea81d545d43f8a9b717 fast sync tip=11613801 pending block hashes=860
[2021-08-18 05:48:03]  INFO [request] Sync progress current chain tip=0x772ece8d90f1c5be984fea675cc33c6af4745cae59f32ea81d545d43f8a9b717 fast sync tip=11613801 pending block hashes=891
[2021-08-18 05:48:09]  INFO [request] Sync progress current chain tip=0x772ece8d90f1c5be984fea675cc33c6af4745cae59f32ea81d545d43f8a9b717 fast sync tip=11613801 pending block hashes=922

The version of Theta I'm using is 3.1.0, should I downgrade the code and rebuild? which branch you would recommend me to do that?

My configuration:

# Theta mainnet guardian configuration
storage:
  statePruningRetainedBlocks: 2048
p2p:
  opt: 0
  port: 30001
  seeds: 18.217.234.19:21000,3.16.9.73:21000,3.20.109.241:21000,18.223.165.134:21000,35.184.232.41:21000,35.230.172.8:21000,34.83.204.5:21000
  minNumPeers: 24
  maxNumPeers: 96
  natMapping: true
  maxConnections: 2048
rpc:
  port: 16888
  enabled: true
  address: 0.0.0.0
log:
  levels: "*:info"

[JohnsonCaii]

Tried both 3.1 and 3.0, all will be stuck at the above block height. Only me having this issue? quite confused.