Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[accounts-db] increase buffer size to improve compression #3446

Open
ksolana opened this issue Nov 2, 2024 · 0 comments
Open

[accounts-db] increase buffer size to improve compression #3446

ksolana opened this issue Nov 2, 2024 · 0 comments

Comments

@ksolana
Copy link

ksolana commented Nov 2, 2024

Problem

hail-is/hail#14033 suggests that increasing the buffer size can reduce the compressed binary size.

Interesting paragraph from: https://engineering.fb.com/2018/12/19/core-infra/zstandard/

In contrast, zstd can effectively make use of megabytes of history (and even more when explicitly requested). With zstd, increasing the block size beyond 256 KB generates significant incremental compression benefits. However, other forces prevent blocks from scaling to arbitrary sizes. Large blocks consume more memory during handling. And random access reads, while not the bulk of the workload, require decompressing the whole block to access an individual element. So it’s useful to keep the block size below a certain threshold. Nevertheless, this is an easily accessible configuration parameter and can now be tuned as needed to produce additional benefits.

Proposed Solution

Increase the buffer size if this is true.
A patch in zstd-rs that provides confugurable buffer size. It is still pending review but we can create a wrapper around our use case as well. gyscos/zstd-rs#300

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants