BlockTableCache ReplayLog Consumes a lot of Memory

[langrp]

Expected behavior

I'm running RocksDB from Java app that consumes data from Kafka, validates and stores them in RocksDB. I use LRU block cache, but total used memory is way beyond what I'd expected. I got jemalloc heap profile to see where the memory goes to. It seems to be consumed by appending string in ReplyLog.

My setup has:

• 2 column family each with 8 memtables by 64MB, which would be 1GB
• Shared block cache with 4GB
• Indexes stored in block cache

Actual behavior

I'd expect about roughly 6GB or so. But the memory grows even up to 20GB if requested (running in k8s).

Steps to reproduce the behavior

prefix_extractor=rocksdb.CappedPrefix.18
compression_per_level=kNoCompression:kNoCompression:kZSTD:kZSTD:kZSTD
memtable_whole_key_filtering=true
target_file_size_multiplier=8
max_write_buffer_number=8
blob_compression_type=kNoCompression
compression=kZSTD
level0_stop_writes_trigger=32
level0_slowdown_writes_trigger=16
strict_max_successive_merges=false
level0_file_num_compaction_trigger=8
memtable_prefix_bloom_size_ratio=0.100000
max_write_buffer_number_to_maintain=8

[TableOptions/BlockBasedTable "default"]
num_file_reads_for_auto_readahead=2
initial_auto_readahead_size=8192
metadata_cache_options={unpartitioned_pinning=kFallback;partition_pinning=kFallback;top_level_index_pinning=kFallback;}
enable_index_compression=true
pin_top_level_index_and_filter=true
read_amp_bytes_per_bit=0
verify_compression=false
prepopulate_block_cache=kDisable
format_version=6
use_delta_encoding=true
partition_filters=true
metadata_block_size=4096
max_auto_readahead_size=262144
index_block_restart_interval=1
block_size_deviation=10
block_size=16384
detect_filter_construct_corruption=false
no_block_cache=false
checksum=kXXH3
filter_policy=bloomfilter:10:false
data_block_hash_table_util_ratio=0.750000
block_restart_interval=16
index_type=kTwoLevelIndexSearch
pin_l0_filter_and_index_blocks_in_cache=true
data_block_index_type=kDataBlockBinarySearch
cache_index_and_filter_blocks_with_high_priority=true
whole_key_filtering=true
index_shortening=kShortenSeparators
cache_index_and_filter_blocks=true
block_align=false
optimize_filters_for_memory=true
flush_block_policy_factory=FlushBlockBySizePolicyFactory