add class to build a compressed block incrementally

[arvidn]

purpose

This PR instroduces a type called BlockBuilder which lets you add spend bundles, one at a time, until the block cost reaches the limit. As spend bundles are added, they are serialized into the final block generator with back-references (deduplication). This makes the final byte-cost difficult to predict, hence the iterative approach.

This patch reproduces some logic from the current mempool block building function, such as the logic of when to give up adding more spend bundles to the block.

cost

When adding a spend bundle you also have to specify the cost of it. This cost is computed when the mempool item is first validated. In existing block creation logic, we use the total cost of the spend bundle, execution cost, condition cost and byte-cost. However, with the BlockBuilder, the accompanied cost is supposed to only be execution cost + condition cost. This is because the byte-cost is only known after the spend bundle has been added.

test

The test picks random permutations of the spend bundles in the test-bundles directory and creates blocks by feeding the bundles one-by-one. The test tracks how long it takes to fill the block, and how long the most expensive call to add_spend_bundle() took.

We need to compute the cost of each spend bundle, which is done on start-up, when all bundles are being loaded.

The test then parses the resulting block and runs it through run_block_generator2() (just like consensus). This ensures the signature is correct, and that the cost is correct.

See the example output below, run on Ubuntu/Threadripper:

loading spend bundles from disk
loaded 90 spend bundles
idx:   0 built block in 11.75 seconds, cost: 10998575664 skipped:  3 longest-call:  9.99s TX: 37
idx:   1 built block in 10.05 seconds, cost: 10983357358 skipped:  7 longest-call:  8.93s TX: 32
idx:   2 built block in 10.88 seconds, cost: 10847563861 skipped:  7 longest-call:  9.50s TX: 62
idx:   3 built block in  2.85 seconds, cost: 10965747249 skipped:  7 longest-call:  0.76s TX: 54
idx:   4 built block in 11.45 seconds, cost: 10754122135 skipped:  7 longest-call:  9.69s TX: 40
idx:   5 built block in  3.31 seconds, cost: 10583555007 skipped:  7 longest-call:  0.88s TX: 46
idx:   6 built block in  9.54 seconds, cost: 10993197686 skipped:  7 longest-call:  8.30s TX: 55
idx:   7 built block in 19.23 seconds, cost: 10977464946 skipped:  7 longest-call: 11.16s TX: 40
idx:   8 built block in 10.08 seconds, cost: 10758545088 skipped:  7 longest-call:  5.90s TX: 39
idx:   9 built block in 10.68 seconds, cost: 10889601079 skipped:  7 longest-call:  8.43s TX: 36
idx:  10 built block in 11.08 seconds, cost: 10996639192 skipped:  3 longest-call: 10.00s TX: 29
idx:  11 built block in 11.44 seconds, cost: 10970637745 skipped:  7 longest-call:  9.22s TX: 34
idx:  12 built block in 11.48 seconds, cost: 10996973255 skipped:  6 longest-call:  8.99s TX: 37
idx:  13 built block in  8.75 seconds, cost: 10999105794 skipped:  6 longest-call:  7.30s TX: 31
idx:  14 built block in 19.63 seconds, cost: 10994129896 skipped:  6 longest-call:  9.26s TX: 40
idx:  15 built block in 11.53 seconds, cost: 10664917802 skipped:  7 longest-call:  8.81s TX: 55
idx:  16 built block in 12.19 seconds, cost: 10994836257 skipped:  6 longest-call: 10.36s TX: 56
idx:  17 built block in  1.93 seconds, cost: 10913252607 skipped:  7 longest-call:  0.34s TX: 42
idx:  18 built block in  2.22 seconds, cost: 10990565458 skipped:  7 longest-call:  0.56s TX: 51
idx:  19 built block in  9.36 seconds, cost: 10997538040 skipped:  3 longest-call:  8.60s TX: 28
idx:  20 built block in 10.04 seconds, cost: 10995570352 skipped:  0 longest-call:  9.38s TX: 23
idx:  21 built block in  9.24 seconds, cost: 10793651464 skipped:  7 longest-call:  6.58s TX: 61
idx:  22 built block in 19.62 seconds, cost: 10996436556 skipped:  4 longest-call: 10.26s TX: 27
idx:  23 built block in 11.23 seconds, cost: 10741182550 skipped:  7 longest-call:  9.00s TX: 47
idx:  24 built block in 18.68 seconds, cost: 10996512566 skipped:  4 longest-call: 11.17s TX: 35
idx:  25 built block in  7.65 seconds, cost: 10511794612 skipped:  7 longest-call:  5.78s TX: 31
idx:  26 built block in  2.80 seconds, cost: 10880611048 skipped:  7 longest-call:  0.61s TX: 55
idx:  27 built block in 10.76 seconds, cost: 10761481781 skipped:  7 longest-call:  9.60s TX: 50
idx:  28 built block in 10.80 seconds, cost: 10700285540 skipped:  7 longest-call:  9.81s TX: 33
idx:  29 built block in  9.47 seconds, cost: 10999810076 skipped:  4 longest-call:  6.80s TX: 32
test gen::build_compressed_block::tests::test_build_block ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 2250 filtered out; finished in 20.55s

Some of these timings are not great. There's separate work in progress to optimize the CLVM compression.

test data

This PR adds some test spend bundles to the test-bundles directory. My intention is to, in a future update, make CI pull down additional test bundles ( Chia-Network/test-cache#16 ) to run more comprehensive tests.

$ ls -la test-bundles/ | wc -l
96
$ du -sh test-bundles/
1.5M    test-bundles/

[coveralls-official]

Pull Request Test Coverage Report for Build 12884465058

Details

• 251 of 262 (95.8%) changed or added relevant lines in 2 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage increased (+0.2%) to 84.611%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
crates/chia-consensus/src/gen/build_compressed_block.rs	250	261	95.79%

Totals
Change from base Build 12884456010:	0.2%
Covered Lines:	13421
Relevant Lines:	15862

---

💛 - Coveralls