You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Originally posted by bartek-siudeja November 22, 2024
kvrocks is quite impressive! Thanks for doing this.
We are trying to build a KV store with extremely strict latency/throughput requirements. We are working with really large servers, with lots of memory. Our values are relatively big and keys are random and small. So it would make sense to store all indexes and big bloom filters in memory. It seems this can be achieved using cache_index_and_filter_blocks: no and max_open_files: -1. However, kvrocks enables partitioning (at least for filters, not sure about indexes): https://github.com/apache/kvrocks/blob/2.10/src/storage/storage.cc#L132
And partitioned filters seem to always live inside block_cache, so they can be evicted.
Would it be possible to expose the partitioning option in the config file? Or maybe have a "meta option" to force filters/indexes to stay in memory. Along the same line bloom filter size could be increased from 10 to say 15 bits to really avoid hitting the disk at any cost. Really only when we are pretty much sure key is there, and value can be retrieved. But this option is also hardcoded (a line above filter partitioning).
The text was updated successfully, but these errors were encountered:
Discussed in #2675
Originally posted by bartek-siudeja November 22, 2024
kvrocks is quite impressive! Thanks for doing this.
We are trying to build a KV store with extremely strict latency/throughput requirements. We are working with really large servers, with lots of memory. Our values are relatively big and keys are random and small. So it would make sense to store all indexes and big bloom filters in memory. It seems this can be achieved using
cache_index_and_filter_blocks: no
andmax_open_files: -1
. However, kvrocks enables partitioning (at least for filters, not sure about indexes):https://github.com/apache/kvrocks/blob/2.10/src/storage/storage.cc#L132
And partitioned filters seem to always live inside block_cache, so they can be evicted.
Would it be possible to expose the partitioning option in the config file? Or maybe have a "meta option" to force filters/indexes to stay in memory. Along the same line bloom filter size could be increased from 10 to say 15 bits to really avoid hitting the disk at any cost. Really only when we are pretty much sure key is there, and value can be retrieved. But this option is also hardcoded (a line above filter partitioning).
The text was updated successfully, but these errors were encountered: