Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow to disable the index partitioning in kvrocks.conf #2676

Open
git-hulk opened this issue Nov 22, 2024 Discussed in #2675 · 0 comments
Open

Allow to disable the index partitioning in kvrocks.conf #2676

git-hulk opened this issue Nov 22, 2024 Discussed in #2675 · 0 comments
Labels
feature type new feature good first issue Good for newcomers

Comments

@git-hulk
Copy link
Member

Discussed in #2675

Originally posted by bartek-siudeja November 22, 2024
kvrocks is quite impressive! Thanks for doing this.

We are trying to build a KV store with extremely strict latency/throughput requirements. We are working with really large servers, with lots of memory. Our values are relatively big and keys are random and small. So it would make sense to store all indexes and big bloom filters in memory. It seems this can be achieved using cache_index_and_filter_blocks: no and max_open_files: -1. However, kvrocks enables partitioning (at least for filters, not sure about indexes):
https://github.com/apache/kvrocks/blob/2.10/src/storage/storage.cc#L132
And partitioned filters seem to always live inside block_cache, so they can be evicted.

Would it be possible to expose the partitioning option in the config file? Or maybe have a "meta option" to force filters/indexes to stay in memory. Along the same line bloom filter size could be increased from 10 to say 15 bits to really avoid hitting the disk at any cost. Really only when we are pretty much sure key is there, and value can be retrieved. But this option is also hardcoded (a line above filter partitioning).

@git-hulk git-hulk added good first issue Good for newcomers feature type new feature labels Nov 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature type new feature good first issue Good for newcomers
Projects
None yet
Development

No branches or pull requests

1 participant