Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GC non pruned states #4526

Closed
wants to merge 4 commits into from
Closed

GC non pruned states #4526

wants to merge 4 commits into from

Conversation

dapplion
Copy link
Contributor

@dapplion dapplion commented Sep 9, 2022

Motivation

For Prater/Goerli nodes running for a few months that extra space accounted for 50GB / 120GB of db data pre-pruning.

I would be nice to automatically prune the existing states since it's very cheap and reliable to detect the non-pruned states.

Description

Once on start-up, read the archive_states bucket keys to check if there are states that should be pruned. If there are:

  • Remove them all in batches
  • Trigger compactRange on archive_states bucket. If this is not done the total db size does not decrease. See for details on some experiments Trigger level db compactRange #4515

marking as draft as test of this branch has caused issues in nodes, still unknown

Sep-07 14:02:22.367[eth1]            error: Error updating eth1 chain cache  Database is not open
Error: Database is not open
    at ClassicLevel.values (/usr/src/lodestar/node_modules/abstract-level/abstract-level.js:726:13)
    at LevelDbController.values (file:///usr/src/lodestar/packages/db/src/controller/level.ts:134:36)
    at DepositEventRepository.values (file:///usr/src/lodestar/packages/db/src/abstractRepository.ts:161:32)
    at DepositEventRepository.lastValue (file:///usr/src/lodestar/packages/db/src/abstractRepository.ts:226:31)
    at Eth1DepositsCache.add (file:///usr/src/lodestar/packages/beacon-node/src/eth1/eth1DepositsCache.ts:44:48)
    at Eth1DepositDataTracker.updateDepositCache (file:///usr/src/lodestar/packages/beacon-node/src/eth1/eth1DepositDataTracker.ts:211:30)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at Eth1DepositDataTracker.update (file:///usr/src/lodestar/packages/beacon-node/src/eth1/eth1DepositDataTracker.ts:191:33)
    at Eth1DepositDataTracker.runAutoUpdate (file:///usr/src/lodestar/packages/beacon-node/src/eth1/eth1DepositDataTracker.ts:154:29)

From #4508 (comment)

@github-actions
Copy link
Contributor

github-actions bot commented Sep 9, 2022

Performance Report

✔️ no performance regression detected

Full benchmark results
Benchmark suite Current: 73034a1 Previous: ac755d9 Ratio
getPubkeys - index2pubkey - req 1000 vs - 250000 vc 2.0825 ms/op 1.8254 ms/op 1.14
getPubkeys - validatorsArr - req 1000 vs - 250000 vc 67.028 us/op 64.583 us/op 1.04
BLS verify - blst-native 2.1647 ms/op 2.1636 ms/op 1.00
BLS verifyMultipleSignatures 3 - blst-native 4.4853 ms/op 4.4772 ms/op 1.00
BLS verifyMultipleSignatures 8 - blst-native 9.7204 ms/op 9.6675 ms/op 1.01
BLS verifyMultipleSignatures 32 - blst-native 35.218 ms/op 35.144 ms/op 1.00
BLS aggregatePubkeys 32 - blst-native 46.462 us/op 46.462 us/op 1.00
BLS aggregatePubkeys 128 - blst-native 182.14 us/op 182.29 us/op 1.00
getAttestationsForBlock 158.47 ms/op 151.08 ms/op 1.05
isKnown best case - 1 super set check 469.00 ns/op 480.00 ns/op 0.98
isKnown normal case - 2 super set checks 456.00 ns/op 470.00 ns/op 0.97
isKnown worse case - 16 super set checks 454.00 ns/op 467.00 ns/op 0.97
CheckpointStateCache - add get delete 9.0720 us/op 8.9150 us/op 1.02
validate gossip signedAggregateAndProof - struct 5.0279 ms/op 5.0262 ms/op 1.00
validate gossip attestation - struct 2.3648 ms/op 2.3852 ms/op 0.99
pickEth1Vote - no votes 2.1272 ms/op 2.1516 ms/op 0.99
pickEth1Vote - max votes 20.443 ms/op 18.824 ms/op 1.09
pickEth1Vote - Eth1Data hashTreeRoot value x2048 13.073 ms/op 13.265 ms/op 0.99
pickEth1Vote - Eth1Data hashTreeRoot tree x2048 21.819 ms/op 21.634 ms/op 1.01
pickEth1Vote - Eth1Data fastSerialize value x2048 1.5818 ms/op 1.5836 ms/op 1.00
pickEth1Vote - Eth1Data fastSerialize tree x2048 13.966 ms/op 13.093 ms/op 1.07
bytes32 toHexString 1.0710 us/op 976.00 ns/op 1.10
bytes32 Buffer.toString(hex) 831.00 ns/op 790.00 ns/op 1.05
bytes32 Buffer.toString(hex) from Uint8Array 1.0730 us/op 1.0460 us/op 1.03
bytes32 Buffer.toString(hex) + 0x 837.00 ns/op 808.00 ns/op 1.04
Object access 1 prop 0.38900 ns/op 0.39200 ns/op 0.99
Map access 1 prop 0.30600 ns/op 0.31500 ns/op 0.97
Object get x1000 12.695 ns/op 10.900 ns/op 1.16
Map get x1000 1.0080 ns/op 1.0990 ns/op 0.92
Object set x1000 82.954 ns/op 72.958 ns/op 1.14
Map set x1000 52.260 ns/op 48.857 ns/op 1.07
Return object 10000 times 0.44140 ns/op 0.44070 ns/op 1.00
Throw Error 10000 times 5.9418 us/op 5.9723 us/op 0.99
enrSubnets - fastDeserialize 64 bits 2.8320 us/op 2.4940 us/op 1.14
enrSubnets - ssz BitVector 64 bits 785.00 ns/op 716.00 ns/op 1.10
enrSubnets - fastDeserialize 4 bits 409.00 ns/op 360.00 ns/op 1.14
enrSubnets - ssz BitVector 4 bits 792.00 ns/op 738.00 ns/op 1.07
prioritizePeers score -10:0 att 32-0.1 sync 2-0 90.576 us/op 82.147 us/op 1.10
prioritizePeers score 0:0 att 32-0.25 sync 2-0.25 115.35 us/op 115.11 us/op 1.00
prioritizePeers score 0:0 att 32-0.5 sync 2-0.5 212.22 us/op 195.56 us/op 1.09
prioritizePeers score 0:0 att 64-0.75 sync 4-0.75 407.75 us/op 382.31 us/op 1.07
prioritizePeers score 0:0 att 64-1 sync 4-1 405.56 us/op 407.10 us/op 1.00
RateTracker 1000000 limit, 1 obj count per request 189.44 ns/op 182.78 ns/op 1.04
RateTracker 1000000 limit, 2 obj count per request 140.78 ns/op 133.01 ns/op 1.06
RateTracker 1000000 limit, 4 obj count per request 113.76 ns/op 108.43 ns/op 1.05
RateTracker 1000000 limit, 8 obj count per request 105.09 ns/op 98.759 ns/op 1.06
RateTracker with prune 4.2700 us/op 4.1090 us/op 1.04
array of 16000 items push then shift 51.593 us/op 51.613 us/op 1.00
LinkedList of 16000 items push then shift 12.689 ns/op 12.569 ns/op 1.01
array of 16000 items push then pop 208.34 ns/op 213.45 ns/op 0.98
LinkedList of 16000 items push then pop 12.313 ns/op 12.123 ns/op 1.02
array of 24000 items push then shift 77.359 us/op 77.349 us/op 1.00
LinkedList of 24000 items push then shift 13.385 ns/op 12.956 ns/op 1.03
array of 24000 items push then pop 199.34 ns/op 196.41 ns/op 1.01
LinkedList of 24000 items push then pop 12.413 ns/op 12.247 ns/op 1.01
intersect bitArray bitLen 8 10.711 ns/op 10.846 ns/op 0.99
intersect array and set length 8 141.74 ns/op 133.58 ns/op 1.06
intersect bitArray bitLen 128 55.526 ns/op 55.666 ns/op 1.00
intersect array and set length 128 1.9064 us/op 1.7903 us/op 1.06
Buffer.concat 32 items 2.0220 ns/op 2.0250 ns/op 1.00
pass gossip attestations to forkchoice per slot 3.0672 ms/op 3.0652 ms/op 1.00
computeDeltas 3.2675 ms/op 3.6153 ms/op 0.90
computeProposerBoostScoreFromBalances 808.85 us/op 803.84 us/op 1.01
altair processAttestation - 250000 vs - 7PWei normalcase 3.7964 ms/op 3.3991 ms/op 1.12
altair processAttestation - 250000 vs - 7PWei worstcase 5.6622 ms/op 5.1740 ms/op 1.09
altair processAttestation - setStatus - 1/6 committees join 184.22 us/op 180.58 us/op 1.02
altair processAttestation - setStatus - 1/3 committees join 360.49 us/op 354.19 us/op 1.02
altair processAttestation - setStatus - 1/2 committees join 513.14 us/op 500.06 us/op 1.03
altair processAttestation - setStatus - 2/3 committees join 668.29 us/op 657.56 us/op 1.02
altair processAttestation - setStatus - 4/5 committees join 931.97 us/op 921.26 us/op 1.01
altair processAttestation - setStatus - 100% committees join 1.1342 ms/op 1.1140 ms/op 1.02
altair processBlock - 250000 vs - 7PWei normalcase 26.815 ms/op 24.124 ms/op 1.11
altair processBlock - 250000 vs - 7PWei normalcase hashState 34.419 ms/op 38.249 ms/op 0.90
altair processBlock - 250000 vs - 7PWei worstcase 89.303 ms/op 82.245 ms/op 1.09
altair processBlock - 250000 vs - 7PWei worstcase hashState 101.88 ms/op 111.64 ms/op 0.91
phase0 processBlock - 250000 vs - 7PWei normalcase 3.5791 ms/op 3.2540 ms/op 1.10
phase0 processBlock - 250000 vs - 7PWei worstcase 52.086 ms/op 50.665 ms/op 1.03
altair processEth1Data - 250000 vs - 7PWei normalcase 763.28 us/op 690.64 us/op 1.11
Tree 40 250000 create 713.52 ms/op 707.85 ms/op 1.01
Tree 40 250000 get(125000) 232.75 ns/op 264.59 ns/op 0.88
Tree 40 250000 set(125000) 2.1660 us/op 2.2053 us/op 0.98
Tree 40 250000 toArray() 27.954 ms/op 27.183 ms/op 1.03
Tree 40 250000 iterate all - toArray() + loop 28.420 ms/op 27.685 ms/op 1.03
Tree 40 250000 iterate all - get(i) 118.38 ms/op 110.62 ms/op 1.07
MutableVector 250000 create 14.888 ms/op 12.510 ms/op 1.19
MutableVector 250000 get(125000) 10.621 ns/op 10.888 ns/op 0.98
MutableVector 250000 set(125000) 548.47 ns/op 532.57 ns/op 1.03
MutableVector 250000 toArray() 6.6104 ms/op 6.0997 ms/op 1.08
MutableVector 250000 iterate all - toArray() + loop 6.7712 ms/op 6.0809 ms/op 1.11
MutableVector 250000 iterate all - get(i) 2.7256 ms/op 2.7632 ms/op 0.99
Array 250000 create 6.4647 ms/op 5.8296 ms/op 1.11
Array 250000 clone - spread 3.2837 ms/op 3.1366 ms/op 1.05
Array 250000 get(125000) 1.5290 ns/op 1.4370 ns/op 1.06
Array 250000 set(125000) 1.5090 ns/op 1.4720 ns/op 1.03
Array 250000 iterate all - loop 150.95 us/op 151.00 us/op 1.00
effectiveBalanceIncrements clone Uint8Array 300000 50.261 us/op 137.46 us/op 0.37
effectiveBalanceIncrements clone MutableVector 300000 1.0800 us/op 640.00 ns/op 1.69
effectiveBalanceIncrements rw all Uint8Array 300000 247.17 us/op 247.07 us/op 1.00
effectiveBalanceIncrements rw all MutableVector 300000 174.05 ms/op 131.75 ms/op 1.32
phase0 afterProcessEpoch - 250000 vs - 7PWei 193.77 ms/op 189.15 ms/op 1.02
phase0 beforeProcessEpoch - 250000 vs - 7PWei 74.800 ms/op 57.362 ms/op 1.30
altair processEpoch - mainnet_e81889 552.90 ms/op 574.62 ms/op 0.96
mainnet_e81889 - altair beforeProcessEpoch 133.24 ms/op 81.213 ms/op 1.64
mainnet_e81889 - altair processJustificationAndFinalization 18.163 us/op 21.123 us/op 0.86
mainnet_e81889 - altair processInactivityUpdates 8.9575 ms/op 9.4378 ms/op 0.95
mainnet_e81889 - altair processRewardsAndPenalties 81.303 ms/op 126.80 ms/op 0.64
mainnet_e81889 - altair processRegistryUpdates 2.7580 us/op 4.2820 us/op 0.64
mainnet_e81889 - altair processSlashings 714.00 ns/op 1.2270 us/op 0.58
mainnet_e81889 - altair processEth1DataReset 681.00 ns/op 1.1550 us/op 0.59
mainnet_e81889 - altair processEffectiveBalanceUpdates 2.1931 ms/op 2.4479 ms/op 0.90
mainnet_e81889 - altair processSlashingsReset 4.4500 us/op 6.6480 us/op 0.67
mainnet_e81889 - altair processRandaoMixesReset 4.2830 us/op 6.3480 us/op 0.67
mainnet_e81889 - altair processHistoricalRootsUpdate 721.00 ns/op 994.00 ns/op 0.73
mainnet_e81889 - altair processParticipationFlagUpdates 2.1450 us/op 2.6100 us/op 0.82
mainnet_e81889 - altair processSyncCommitteeUpdates 609.00 ns/op 833.00 ns/op 0.73
mainnet_e81889 - altair afterProcessEpoch 202.30 ms/op 200.94 ms/op 1.01
phase0 processEpoch - mainnet_e58758 492.82 ms/op 494.43 ms/op 1.00
mainnet_e58758 - phase0 beforeProcessEpoch 187.57 ms/op 188.67 ms/op 0.99
mainnet_e58758 - phase0 processJustificationAndFinalization 17.546 us/op 21.146 us/op 0.83
mainnet_e58758 - phase0 processRewardsAndPenalties 120.74 ms/op 71.354 ms/op 1.69
mainnet_e58758 - phase0 processRegistryUpdates 7.9190 us/op 10.478 us/op 0.76
mainnet_e58758 - phase0 processSlashings 618.00 ns/op 1.3330 us/op 0.46
mainnet_e58758 - phase0 processEth1DataReset 687.00 ns/op 800.00 ns/op 0.86
mainnet_e58758 - phase0 processEffectiveBalanceUpdates 1.6737 ms/op 1.9315 ms/op 0.87
mainnet_e58758 - phase0 processSlashingsReset 3.7710 us/op 4.8330 us/op 0.78
mainnet_e58758 - phase0 processRandaoMixesReset 4.2080 us/op 5.9760 us/op 0.70
mainnet_e58758 - phase0 processHistoricalRootsUpdate 735.00 ns/op 1.0230 us/op 0.72
mainnet_e58758 - phase0 processParticipationRecordUpdates 3.6070 us/op 5.3320 us/op 0.68
mainnet_e58758 - phase0 afterProcessEpoch 166.31 ms/op 165.05 ms/op 1.01
phase0 processEffectiveBalanceUpdates - 250000 normalcase 1.9396 ms/op 2.0652 ms/op 0.94
phase0 processEffectiveBalanceUpdates - 250000 worstcase 0.5 2.2176 ms/op 2.5194 ms/op 0.88
altair processInactivityUpdates - 250000 normalcase 40.119 ms/op 33.386 ms/op 1.20
altair processInactivityUpdates - 250000 worstcase 33.517 ms/op 40.137 ms/op 0.84
phase0 processRegistryUpdates - 250000 normalcase 6.5580 us/op 9.2880 us/op 0.71
phase0 processRegistryUpdates - 250000 badcase_full_deposits 378.81 us/op 375.10 us/op 1.01
phase0 processRegistryUpdates - 250000 worstcase 0.5 194.67 ms/op 182.03 ms/op 1.07
altair processRewardsAndPenalties - 250000 normalcase 71.669 ms/op 104.28 ms/op 0.69
altair processRewardsAndPenalties - 250000 worstcase 98.597 ms/op 108.90 ms/op 0.91
phase0 getAttestationDeltas - 250000 normalcase 12.160 ms/op 11.958 ms/op 1.02
phase0 getAttestationDeltas - 250000 worstcase 13.105 ms/op 12.124 ms/op 1.08
phase0 processSlashings - 250000 worstcase 5.0442 ms/op 5.2024 ms/op 0.97
altair processSyncCommitteeUpdates - 250000 301.20 ms/op 301.94 ms/op 1.00
BeaconState.hashTreeRoot - No change 545.00 ns/op 563.00 ns/op 0.97
BeaconState.hashTreeRoot - 1 full validator 74.475 us/op 71.903 us/op 1.04
BeaconState.hashTreeRoot - 32 full validator 752.66 us/op 650.60 us/op 1.16
BeaconState.hashTreeRoot - 512 full validator 7.0660 ms/op 7.4920 ms/op 0.94
BeaconState.hashTreeRoot - 1 validator.effectiveBalance 91.210 us/op 90.327 us/op 1.01
BeaconState.hashTreeRoot - 32 validator.effectiveBalance 1.3482 ms/op 1.3303 ms/op 1.01
BeaconState.hashTreeRoot - 512 validator.effectiveBalance 17.489 ms/op 17.578 ms/op 0.99
BeaconState.hashTreeRoot - 1 balances 72.891 us/op 69.135 us/op 1.05
BeaconState.hashTreeRoot - 32 balances 744.92 us/op 660.03 us/op 1.13
BeaconState.hashTreeRoot - 512 balances 6.3135 ms/op 6.4038 ms/op 0.99
BeaconState.hashTreeRoot - 250000 balances 99.764 ms/op 100.25 ms/op 1.00
aggregationBits - 2048 els - zipIndexesInBitList 31.086 us/op 25.137 us/op 1.24
regular array get 100000 times 60.560 us/op 64.141 us/op 0.94
wrappedArray get 100000 times 60.581 us/op 60.862 us/op 1.00
arrayWithProxy get 100000 times 28.317 ms/op 29.011 ms/op 0.98
ssz.Root.equals 535.00 ns/op 485.00 ns/op 1.10
byteArrayEquals 523.00 ns/op 477.00 ns/op 1.10
shuffle list - 16384 els 11.823 ms/op 11.336 ms/op 1.04
shuffle list - 250000 els 171.85 ms/op 169.62 ms/op 1.01
processSlot - 1 slots 15.301 us/op 13.421 us/op 1.14
processSlot - 32 slots 1.9723 ms/op 2.1377 ms/op 0.92
getEffectiveBalanceIncrementsZeroInactive - 250000 vs - 7PWei 429.99 us/op 484.68 us/op 0.89
getCommitteeAssignments - req 1 vs - 250000 vc 5.3973 ms/op 5.4002 ms/op 1.00
getCommitteeAssignments - req 100 vs - 250000 vc 7.9044 ms/op 7.9028 ms/op 1.00
getCommitteeAssignments - req 1000 vs - 250000 vc 8.5044 ms/op 8.4658 ms/op 1.00
RootCache.getBlockRootAtSlot - 250000 vs - 7PWei 7.8300 ns/op 8.5900 ns/op 0.91
state getBlockRootAtSlot - 250000 vs - 7PWei 1.2149 us/op 1.1159 us/op 1.09
computeProposers - vc 250000 18.136 ms/op 18.261 ms/op 0.99
computeEpochShuffling - vc 250000 175.25 ms/op 173.05 ms/op 1.01
getNextSyncCommittee - vc 250000 299.16 ms/op 297.85 ms/op 1.00

by benchmarkbot/action

@dapplion
Copy link
Contributor Author

Closing as it's not worth the complexity. The original issue was fixed long ago so most users do not need this retroactive cleanup

@dapplion dapplion closed this May 24, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@dapplion