Trigger level db compactRange

[dapplion]

• From FIx archive finalized states db entry key typo #4508 (comment)

Due to level-db internals, after a significant amount of data put or del it's necessary to compact that range to realize the size reduction. After an offline discussion with other teams:

• Should compact all archived states range after the prune of this PR
• Should compact after completing sync
• Should consider compacting the DB at most once per 7 days

Given we have not done this before, extensive testing is necessary to:

• Benchmark CPU and time cost of compactRange on different key ranges, plus how much size is free'ed on average situtations
• Understand the impact on performance on the beacon node if any. While NodeJS is mostly single thread, I would expect this ops to happen in the C++ worker thread and not mess with validator performance

[dapplion]

I've added a new option on a branch dapplion/bump-level...dapplion/test-compact which triggers compact of the entire db on start up with --compactDbOnStartup. I've deployed this branch in all feat2 hosts as of 11:20 CEST Sept 7th

Results for feat2-sm1v-ctvpss

• node_filesystem_avail_bytes changed from 68GB -> 121GB: Total freed 53GB
• Total time 13 min

Sep-07 11:27:00.863[]                 info: db compactRange on byte 0/256 allForks_stateArchive...
Sep-07 11:33:00.127[]                 info: db compactRange on byte 1/256 allForks_block...
Sep-07 11:33:02.192[]                 info: db compactRange on byte 2/256 allForks_blockArchive...
Sep-07 11:40:06.486[]                 info: db compactRange on byte 3/256 index_blockArchiveParentRootIndex...
Sep-07 11:40:11.251[]                 info: db compactRange on byte 4/256 index_blockArchiveRootIndex...
Sep-07 11:40:16.204[]                 info: db compactRange on byte 6/256 index_mainChain...
Sep-07 11:40:16.239[]                 info: db compactRange on byte 7/256 index_chainInfo...
Sep-07 11:40:16.273[]                 info: db compactRange on byte 8/256 phase0_eth1Data...
Sep-07 11:40:16.792[]                 info: db compactRange on byte 9/256 index_depositDataRoot...
Sep-07 11:40:16.897[]                 info: db compactRange on byte 12/256 phase0_depositData...
Sep-07 11:40:16.938[]                 info: db compactRange on byte 13/256 phase0_exit...
Sep-07 11:40:16.972[]                 info: db compactRange on byte 14/256 phase0_proposerSlashing...
Sep-07 11:40:17.009[]                 info: db compactRange on byte 15/256 phase0_attesterSlashing...
Sep-07 11:40:17.167[]                 info: db compactRange on byte 19/256 phase0_depositEvent...
Sep-07 11:40:17.593[]                 info: db compactRange on byte 20/256 phase0_slashingProtectionBlockBySlot...
Sep-07 11:40:17.627[]                 info: db compactRange on byte 21/256 phase0_slashingProtectionAttestationByTarget...
Sep-07 11:40:17.657[]                 info: db compactRange on byte 22/256 phase0_slashingProtectionAttestationLowerBound...
Sep-07 11:40:17.696[]                 info: db compactRange on byte 23/256 index_slashingProtectionMinSpanDistance...
Sep-07 11:40:17.735[]                 info: db compactRange on byte 24/256 index_slashingProtectionMaxSpanDistance...
Sep-07 11:40:17.831[]                 info: db compactRange on byte 26/256 index_stateArchiveRootIndex...
Sep-07 11:40:17.966[]                 info: db compactRange on byte 30/256 phase0_preGenesisState...
Sep-07 11:40:17.998[]                 info: db compactRange on byte 31/256 phase0_preGenesisStateLastProcessedBlock...
Sep-07 11:40:18.416[]                 info: db compactRange on byte 41/256 validator_metaData...
Sep-07 11:40:18.456[]                 info: db compactRange on byte 42/256 backfilled_ranges...
Sep-07 11:40:18.793[]                 info: db compactRange on byte 51/256 lightClient_syncCommitteeWitness...
Sep-07 11:40:19.391[]                 info: db compactRange on byte 52/256 lightClient_syncCommittee...
Sep-07 11:40:19.419[]                 info: db compactRange on byte 53/256 lightClient_checkpointHeader...
Sep-07 11:40:19.813[]                 info: db compactRange on byte 54/256 lightClient_bestPartialLightClientUpdate...
Sep-07 11:40:19.844[]                 info: db compactRange on byte 55/256 -...
...
Sep-07 11:40:27.167[]                 info: db compactRange on byte 254/256 -...
Sep-07 11:40:27.201[]                 info: db compactRange completed

Results for feat2-novc-ctvpss

• node_filesystem_avail_bytes changed from 68.5GB -> 120GB: Total freed 51.5GB
• Total time 12 min

devops@feat2-novc-ctvpss:~$ grep compactRange beacon/beacon-2022-09-07.log
Sep-07 11:27:00.795[]                 info: db compactRange on byte 0/256 allForks_stateArchive...
Sep-07 11:33:10.167[]                 info: db compactRange on byte 1/256 allForks_block...
Sep-07 11:33:11.898[]                 info: db compactRange on byte 2/256 allForks_blockArchive...
Sep-07 11:38:18.848[]                 info: db compactRange on byte 3/256 index_blockArchiveParentRootIndex...
Sep-07 11:38:23.226[]                 info: db compactRange on byte 4/256 index_blockArchiveRootIndex...
Sep-07 11:38:26.930[]                 info: db compactRange on byte 6/256 index_mainChain...
Sep-07 11:38:26.966[]                 info: db compactRange on byte 7/256 index_chainInfo...
Sep-07 11:38:27.005[]                 info: db compactRange on byte 8/256 phase0_eth1Data...
Sep-07 11:38:27.822[]                 info: db compactRange on byte 9/256 index_depositDataRoot...
Sep-07 11:38:27.906[]                 info: db compactRange on byte 12/256 phase0_depositData...
Sep-07 11:38:27.939[]                 info: db compactRange on byte 13/256 phase0_exit...
Sep-07 11:38:27.966[]                 info: db compactRange on byte 14/256 phase0_proposerSlashing...
Sep-07 11:38:27.993[]                 info: db compactRange on byte 15/256 phase0_attesterSlashing...
Sep-07 11:38:28.105[]                 info: db compactRange on byte 19/256 phase0_depositEvent...
Sep-07 11:38:28.240[]                 info: db compactRange on byte 20/256 phase0_slashingProtectionBlockBySlot...
Sep-07 11:38:28.267[]                 info: db compactRange on byte 21/256 phase0_slashingProtectionAttestationByTarget...
Sep-07 11:38:28.298[]                 info: db compactRange on byte 22/256 phase0_slashingProtectionAttestationLowerBound...
Sep-07 11:38:28.327[]                 info: db compactRange on byte 23/256 index_slashingProtectionMinSpanDistance...
Sep-07 11:38:28.354[]                 info: db compactRange on byte 24/256 index_slashingProtectionMaxSpanDistance...
Sep-07 11:38:28.419[]                 info: db compactRange on byte 26/256 index_stateArchiveRootIndex...
Sep-07 11:38:28.532[]                 info: db compactRange on byte 30/256 phase0_preGenesisState...
Sep-07 11:38:28.560[]                 info: db compactRange on byte 31/256 phase0_preGenesisStateLastProcessedBlock...
Sep-07 11:38:28.866[]                 info: db compactRange on byte 41/256 validator_metaData...
Sep-07 11:38:28.891[]                 info: db compactRange on byte 42/256 backfilled_ranges...
Sep-07 11:38:29.192[]                 info: db compactRange on byte 51/256 lightClient_syncCommitteeWitness...
Sep-07 11:38:29.825[]                 info: db compactRange on byte 52/256 lightClient_syncCommittee...
Sep-07 11:38:30.121[]                 info: db compactRange on byte 53/256 lightClient_checkpointHeader...
Sep-07 11:38:30.563[]                 info: db compactRange on byte 54/256 lightClient_bestPartialLightClientUpdate...
Sep-07 11:38:30.589[]                 info: db compactRange on byte 55/256 -...
...
Sep-07 11:38:37.320[]                 info: db compactRange on byte 254/256 -...
Sep-07 11:38:37.351[]                 info: db compactRange completed

Results for feat2-novc-ctvpss

Note this instance did not prune additional states beforehand, the two above prune states but did not compact

• node_filesystem_avail_bytes changed from 128GB -> 128GB: Total freed 0GB
• Total time 90 min

devops@feat2-md16-ctvpsm:~$ grep compactRange beacon/beacon-2022-09-07.log 
Sep-07 11:27:07.681[]                 info: db compactRange on byte 0/256 allForks_stateArchive...
Sep-07 12:26:58.457[]                 info: db compactRange on byte 0/256 allForks_stateArchive...
Sep-07 12:26:58.519[]                 info: db compactRange on byte 1/256 allForks_block...
Sep-07 12:27:00.476[]                 info: db compactRange on byte 2/256 allForks_blockArchive...
Sep-07 12:56:07.042[]                 info: db compactRange on byte 3/256 index_blockArchiveParentRootIndex...
Sep-07 12:57:24.598[]                 info: db compactRange completed

[tmm360]

Any news on this? I'm running lodestar on gnosis, and periodically I have to delete db and redo sync because it is increasing by ~10GB per month.

[dapplion]

Any news on this? I'm running lodestar on gnosis, and periodically I have to delete db and redo sync because it is increasing by ~10GB per month.

This proposal was started to clean-up the consequences of an old bug where Lodestar persisted states too frequently. If you have re-synced Lodestar recently this proposal won't affect much. To reduce disk space we are working on this optimization which should reduce disk requirements significantly. I have to check Gnosis specific numbers, but for Ethereum it could be save 10-30 GB / month

• De-duplicate payload from persisted beacon blocks #5671