Adds bench for scan_pubkeys() #2095

brooksprumo · 2024-07-11T18:41:10Z

Problem

There are no benchmarks for how fast/slow the append vec file io is compared to mmap.

Summary of Changes

Add a benchmark for scan_pubkeys().

We recently pushed a change to optimize this function for file io, but it was not straightforward to know how large the scan buffer should be. This benchmark was used to inform the buffer size.

Example benchmark results

Running the benchmark on a shared dev box, I saw these results from a run:

scan_pubkeys/append_vec_mmap/1
                        time:   [9.5220 ns 9.5265 ns 9.5307 ns]
                        thrpt:  [104.92 Melem/s 104.97 Melem/s 105.02 Melem/s]
scan_pubkeys/append_vec_file/1
                        time:   [864.41 ns 865.25 ns 866.12 ns]
                        thrpt:  [1.1546 Melem/s 1.1557 Melem/s 1.1569 Melem/s]
scan_pubkeys/hot_storage/1
                        time:   [5.0997 ns 5.1031 ns 5.1074 ns]
                        thrpt:  [195.79 Melem/s 195.96 Melem/s 196.09 Melem/s]
scan_pubkeys/append_vec_mmap/100
                        time:   [528.48 ns 528.94 ns 529.42 ns]
                        thrpt:  [188.89 Melem/s 189.06 Melem/s 189.22 Melem/s]
scan_pubkeys/append_vec_file/100
                        time:   [58.098 µs 58.137 µs 58.176 µs]
                        thrpt:  [1.7189 Melem/s 1.7201 Melem/s 1.7212 Melem/s]
scan_pubkeys/hot_storage/100
                        time:   [453.75 ns 453.94 ns 454.14 ns]
                        thrpt:  [220.20 Melem/s 220.29 Melem/s 220.38 Melem/s]
scan_pubkeys/append_vec_mmap/1000
                        time:   [6.0991 µs 6.1027 µs 6.1064 µs]
                        thrpt:  [163.76 Melem/s 163.86 Melem/s 163.96 Melem/s]
scan_pubkeys/append_vec_file/1000
                        time:   [800.59 µs 800.70 µs 800.82 µs]
                        thrpt:  [1.2487 Melem/s 1.2489 Melem/s 1.2491 Melem/s]
scan_pubkeys/hot_storage/1000
                        time:   [4.4630 µs 4.4647 µs 4.4663 µs]
                        thrpt:  [223.90 Melem/s 223.98 Melem/s 224.07 Melem/s]
scan_pubkeys/append_vec_mmap/10000
                        time:   [89.066 µs 89.112 µs 89.167 µs]
                        thrpt:  [112.15 Melem/s 112.22 Melem/s 112.28 Melem/s]
scan_pubkeys/append_vec_file/10000
                        time:   [6.1940 ms 6.1949 ms 6.1958 ms]
                        thrpt:  [1.6140 Melem/s 1.6142 Melem/s 1.6145 Melem/s]
scan_pubkeys/hot_storage/10000
                        time:   [43.545 µs 43.579 µs 43.617 µs]
                        thrpt:  [229.27 Melem/s 229.47 Melem/s 229.65 Melem/s]

HaoranYi · 2024-07-11T21:17:12Z

Nice!
hot-storage performs best, which is expected.
append-vec-file is worst 😑 , 2 orders slower than mmap.
Can you add another axis of benchmark with different buffer sizes for append-vec-file scan?

brooksprumo · 2024-07-11T23:00:08Z

Nice! hot-storage performs best, which is expected. append-vec-file is worst 😑 , 2 orders slower than mmap.

Yep 😥

Can you add another axis of benchmark with different buffer sizes for append-vec-file scan?

Yes. I'll do this in a follow up PR.

brooksprumo · 2024-07-12T14:28:07Z

Can you add another axis of benchmark with different buffer sizes for append-vec-file scan?

Yes. I'll do this in a follow up PR.

I'm having second thoughts on this now. In order to make a benchmark for the different buffer sizes, there are a number of internals we need to split out and make public for the bench. I'm not sure that's really worth it. Instead, we can do our normal A/B testing with performance numbers in PRs to justify changes to the buffer sizes.

With that said, I did write the benchmark to see how it would look, and here are the results:

benchmark results

These are the results on a dev box, with just the File IO numbers. I benchmarked the following buffer sizes:

4 KiB <-- page size
64 KIB
128 KiB
256 KiB <-- current buffer size
1 MiB
10 MiB
~30 MiB <-- buffer size for other scan operations

scan_pubkeys/append_vec_file/accounts_1_buffer_4096
                        time:   [824.57 ns 825.30 ns 825.99 ns]
                        thrpt:  [1.2107 Melem/s 1.2117 Melem/s 1.2127 Melem/s]
scan_pubkeys/append_vec_file/accounts_1_buffer_65536
                        time:   [812.21 ns 812.85 ns 813.44 ns]
                        thrpt:  [1.2294 Melem/s 1.2302 Melem/s 1.2312 Melem/s]
scan_pubkeys/append_vec_file/accounts_1_buffer_131072
                        time:   [832.00 ns 832.60 ns 833.21 ns]
                        thrpt:  [1.2002 Melem/s 1.2011 Melem/s 1.2019 Melem/s]
scan_pubkeys/append_vec_file/accounts_1_buffer_262144
                        time:   [838.02 ns 839.26 ns 840.43 ns]
                        thrpt:  [1.1899 Melem/s 1.1915 Melem/s 1.1933 Melem/s]
scan_pubkeys/append_vec_file/accounts_1_buffer_1048576
                        time:   [831.15 ns 831.91 ns 832.61 ns]
                        thrpt:  [1.2010 Melem/s 1.2020 Melem/s 1.2031 Melem/s]
scan_pubkeys/append_vec_file/accounts_1_buffer_10485760
                        time:   [832.52 ns 833.25 ns 833.95 ns]
                        thrpt:  [1.1991 Melem/s 1.2001 Melem/s 1.2012 Melem/s]
scan_pubkeys/append_vec_file/accounts_1_buffer_31461376
                        time:   [824.33 ns 825.40 ns 826.61 ns]
                        thrpt:  [1.2098 Melem/s 1.2115 Melem/s 1.2131 Melem/s]

scan_pubkeys/append_vec_file/accounts_100_buffer_4096
                        time:   [8.9407 µs 8.9507 µs 8.9604 µs]
                        thrpt:  [11.160 Melem/s 11.172 Melem/s 11.185 Melem/s]
scan_pubkeys/append_vec_file/accounts_100_buffer_65536
                        time:   [12.301 µs 12.307 µs 12.313 µs]
                        thrpt:  [8.1212 Melem/s 8.1253 Melem/s 8.1293 Melem/s]
scan_pubkeys/append_vec_file/accounts_100_buffer_131072
                        time:   [31.828 µs 31.845 µs 31.860 µs]
                        thrpt:  [3.1388 Melem/s 3.1402 Melem/s 3.1419 Melem/s]
scan_pubkeys/append_vec_file/accounts_100_buffer_262144
                        time:   [74.721 µs 74.749 µs 74.776 µs]
                        thrpt:  [1.3373 Melem/s 1.3378 Melem/s 1.3383 Melem/s]
scan_pubkeys/append_vec_file/accounts_100_buffer_1048576
                        time:   [157.64 µs 157.69 µs 157.74 µs]
                        thrpt:  [633.95 Kelem/s 634.16 Kelem/s 634.36 Kelem/s]
scan_pubkeys/append_vec_file/accounts_100_buffer_10485760
                        time:   [1.4630 ms 1.4632 ms 1.4635 ms]
                        thrpt:  [68.330 Kelem/s 68.342 Kelem/s 68.354 Kelem/s]
scan_pubkeys/append_vec_file/accounts_100_buffer_31461376
                        time:   [1.5628 ms 1.5634 ms 1.5641 ms]
                        thrpt:  [63.934 Kelem/s 63.963 Kelem/s 63.988 Kelem/s]

scan_pubkeys/append_vec_file/accounts_1000_buffer_4096
                        time:   [103.76 µs 103.81 µs 103.86 µs]
                        thrpt:  [9.6288 Melem/s 9.6332 Melem/s 9.6373 Melem/s]
scan_pubkeys/append_vec_file/accounts_1000_buffer_65536
                        time:   [220.00 µs 220.11 µs 220.22 µs]
                        thrpt:  [4.5409 Melem/s 4.5432 Melem/s 4.5455 Melem/s]
scan_pubkeys/append_vec_file/accounts_1000_buffer_131072
                        time:   [414.00 µs 414.24 µs 414.61 µs]
                        thrpt:  [2.4119 Melem/s 2.4140 Melem/s 2.4155 Melem/s]
scan_pubkeys/append_vec_file/accounts_1000_buffer_262144
                        time:   [1.0473 ms 1.0476 ms 1.0481 ms]
                        thrpt:  [954.15 Kelem/s 954.52 Kelem/s 954.88 Kelem/s]
scan_pubkeys/append_vec_file/accounts_1000_buffer_1048576
                        time:   [1.9184 ms 1.9189 ms 1.9195 ms]
                        thrpt:  [520.98 Kelem/s 521.13 Kelem/s 521.28 Kelem/s]
scan_pubkeys/append_vec_file/accounts_1000_buffer_10485760
                        time:   [18.095 ms 18.099 ms 18.102 ms]
                        thrpt:  [55.242 Kelem/s 55.253 Kelem/s 55.265 Kelem/s]
scan_pubkeys/append_vec_file/accounts_1000_buffer_31461376
                        time:   [25.365 ms 25.369 ms 25.373 ms]
                        thrpt:  [39.412 Kelem/s 39.418 Kelem/s 39.425 Kelem/s]

scan_pubkeys/append_vec_file/accounts_10000_buffer_4096
                        time:   [1.0762 ms 1.0766 ms 1.0770 ms]
                        thrpt:  [9.2853 Melem/s 9.2886 Melem/s 9.2921 Melem/s]
scan_pubkeys/append_vec_file/accounts_10000_buffer_65536
                        time:   [1.9335 ms 1.9342 ms 1.9348 ms]
                        thrpt:  [5.1685 Melem/s 5.1701 Melem/s 5.1719 Melem/s]
scan_pubkeys/append_vec_file/accounts_10000_buffer_131072
                        time:   [3.7189 ms 3.7200 ms 3.7212 ms]
                        thrpt:  [2.6873 Melem/s 2.6882 Melem/s 2.6890 Melem/s]
scan_pubkeys/append_vec_file/accounts_10000_buffer_262144
                        time:   [6.6761 ms 6.6775 ms 6.6788 ms]
                        thrpt:  [1.4973 Melem/s 1.4976 Melem/s 1.4979 Melem/s]
scan_pubkeys/append_vec_file/accounts_10000_buffer_1048576
                        time:   [14.215 ms 14.218 ms 14.221 ms]
                        thrpt:  [703.21 Kelem/s 703.34 Kelem/s 703.48 Kelem/s]
scan_pubkeys/append_vec_file/accounts_10000_buffer_10485760
                        time:   [145.74 ms 145.77 ms 145.80 ms]
                        thrpt:  [68.585 Kelem/s 68.600 Kelem/s 68.615 Kelem/s]
scan_pubkeys/append_vec_file/accounts_10000_buffer_31461376
                        time:   [187.31 ms 187.33 ms 187.34 ms]
                        thrpt:  [53.378 Kelem/s 53.382 Kelem/s 53.386 Kelem/s]

Probably the most useful results are with 1,000 and 10,000 accounts in the file, as that's most similar to mnb. We see with these tests that the smaller buffers all perform better than the larger buffers.

Even though 4 KiB is the fastest in the benchmark, I'm not sure how this would fare in a real validator running on mnb. Maybe 64 KiB is the right tradeoff?

When I made the buffer size 256 KiB in the previous PR, I only tests 256 KiB, 512 KiB, and ~30 MiB. So it makes sense I saw the fastest times with 256 KiB.

HaoranYi · 2024-07-12T14:47:57Z

In order to make a benchmark for the different buffer sizes, there are a number of internals we need to split out and make public for the bench. I'm not sure that's really worth it.

Agree. No need to commit to the changes for support varing buffer size to master. It is fine to keep them separately in the experimental branch.

HaoranYi · 2024-07-12T14:51:45Z

Even though 4 KiB is the fastest in the benchmark, I'm not sure how this would fare in a real validator running on mnb. Maybe 64 KiB is the right tradeoff?

103.76 µs (4k) vs 220.00 µs (64K) almost 2x.

Yes. The number should be highly dependent on the account's sizes. I am okay with 64KiB.

Maybe we can take some sample storage files from mainnet and benchmark on those storages...

brooksprumo self-assigned this Jul 11, 2024

brooksprumo force-pushed the append-vec/scan-pubkeys-bench branch from 6d05c36 to 7395541 Compare July 11, 2024 18:43

Adds bench for scan_pubkeys()

e6ae343

brooksprumo force-pushed the append-vec/scan-pubkeys-bench branch from 7395541 to e6ae343 Compare July 11, 2024 19:17

brooksprumo marked this pull request as ready for review July 11, 2024 20:14

brooksprumo requested a review from HaoranYi July 11, 2024 20:14

HaoranYi approved these changes Jul 11, 2024

View reviewed changes

brooksprumo merged commit 8db4f56 into anza-xyz:master Jul 11, 2024
40 checks passed

brooksprumo deleted the append-vec/scan-pubkeys-bench branch July 11, 2024 23:00

brooksprumo mentioned this pull request Jul 12, 2024

Changes scan buffer size (without data) to 64 KiB #2106

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adds bench for scan_pubkeys() #2095

Adds bench for scan_pubkeys() #2095

brooksprumo commented Jul 11, 2024

HaoranYi commented Jul 11, 2024 •

edited

Loading

brooksprumo commented Jul 11, 2024

brooksprumo commented Jul 12, 2024

HaoranYi commented Jul 12, 2024 •

edited

Loading

HaoranYi commented Jul 12, 2024 •

edited

Loading

Adds bench for scan_pubkeys() #2095

Adds bench for scan_pubkeys() #2095

Conversation

brooksprumo commented Jul 11, 2024

Problem

Summary of Changes

HaoranYi commented Jul 11, 2024 • edited Loading

brooksprumo commented Jul 11, 2024

brooksprumo commented Jul 12, 2024

HaoranYi commented Jul 12, 2024 • edited Loading

HaoranYi commented Jul 12, 2024 • edited Loading

HaoranYi commented Jul 11, 2024 •

edited

Loading

HaoranYi commented Jul 12, 2024 •

edited

Loading

HaoranYi commented Jul 12, 2024 •

edited

Loading