node configuration for full blockchain history

[mbt-commander]

What configuration settings are needed so that you don't need the snapshot to start up a node and download the full history instead of the pruned? When running the example call for GetBlockByHeight for block height 3 the result set is empty.

call:
curl -X POST -H 'Content-Type: application/json' --data '{"jsonrpc":"2.0","method":"theta.GetBlockByHeight","params":[{"height":"3", "include_eth_tx_hashes":false}],"id":1}' http://localhost:16888/rpc

response:
{"jsonrpc":"2.0","id":1,"result":{}}

[mbt-commander]

some update that are promising, but nothing conclusive yet. Here's what I've figured out so far:

• changed the storage section of config to only have statePruningEnabled: false
• instead of using the snapshot that is in the docs, I'm now using the genesis snapshot found here: walletnode/genesis

node is still syncing after about 6 hours and has reach a block height of 2134. using the rpc call about I can get history for block heights 1-2134 but not anything above that. I did not notice in the log this entry:
INFO [request] Sync progress current chain tip=0xf81dba1cd4f767df57f14c1a906061bc4b67e492c0929393626d71d368acf47f fast sync tip=2136 pending block hashes=11341

The pending block hashes keeps going up, but the fast sync tip stays the same and seems to correlate to the latest_finalized_block_height output in the thetacli query status call. Not sure what all that mean, hoping that at some point it will take the pending blocks and make them available. Looking at the theta process it is writing a good amount of data 750MB so far.

I'll keep posting until I figure this out, maybe will help someone else who wants to run a full node. Would also very much appreciate some help from anyone that might have already set up a full node, have been working on a web3 Theta project for months now and need to have a full node as the Theta Explorer API just won't work for my use case anymore.

[mbt-commander]

This set up appears to be working, but unbelievably slow. Running on gigabit fiber (wired) and a dedicated box with SSD drive and 32GB RAM, it's only finalized ~16500 blocks in 24 hours. At this rate it will take almost 2.5 years to download the full blockchain history.

[mbt-commander]

One thing I noticed was the amount of swap being used and how little RAM was. I started lowering the swappiness value until I finally set it to 0 as I wasn't sure if it was making any difference. Slowly the balance has started to shift from swap to RAM and that appears to be speeding things up. Previously was only finalizing about 16k blocks a day, now doing ~70k / day. Hoping this trend continues and is able to speed things up dramatically. at current rate it would be more like 8 months instead of 2.5 years as previously estimated. Anything under a year would work. As this data has become more valuable based on the difficulty of collecting it I have also implemented a decent rotating backup strategy.

[mbt-commander]

Everything was going well until today. Have finalized ~2.9MM blocks so far, swap and swappiness clearly have an impact on finalization speed but not able to keep it stable ranging from 10K to 80k blocks a day. For some reason the fast sync tip went to 0 and no more blocks are finalizing, no idea what is happening. Going to try rolling back to a back up from 2 days ago and restart to see if that helps.

[mbt-commander]

Rolling back to a previous backup didn't work, when starting up I get an error saying too many open files in one of the rolling directories. So seems my 7 day rolling backup strategy doesn't work as expected which sucks, guess I need to test the usability of an rsync backup taken while the node is running. I then moved the original db directory back into place and restarted. No errors, but fast sync tip was still at 0. It is incrementing now which it didn't before, hoping it will not take long to get back to where I was since all of the blocks had previously been finalized and my db directory has about 100GB of files in it.

[mbt-commander]

So basically the node started over once fast sync tip went to 0 for some unknown reason. Tried a few things, but nothing worked and was left with no other option than deleting the db directory and starting over. Going to do some experiments on creating usable backups, seems that backing up the db directory while the node won't work as some files are left in an open state and are not usable on start up. Going to test stopping the node, rsync the db directory and then start up again. Then move the db backup to another node and try starting up and see what fast sync tip is reporting. Fingers crossed it works as it would be nice to have a reliable back up since it is so painful to acquire Theta blocks!