Expose More Performance-Related Configurations

[coltmcnealy-lh]

No description provided.

[coltmcnealy-lh]

Doing testing with the defaults.

• Core Stream Threads: 2
• Timer Stream Threads: 2
• Core Stream Commit Interval: 5 seconds
• Timer Stream Commit Interval: 30 seconds
• Core Memtable Size: 64MB
• Timer Memtable Size: 32MB
• Core State Store Cache Size: 32MB
• Timer State Store Cache Size: 64MB
• Shared RocksDB Block Cache Size: 64MB
• Write Buffer Manager Size: 256MB

12:54:22 -> for i in $(seq 1 1000); do     lhctl run hundred-tasks; done

It completed in 10 minutes 3 seconds. Yikes. Looked like a lot of RocksDB Write Stalls. Most likely caused by the Write Buffer Manager Size being too small.

[coltmcnealy-lh]

• Core Stream Threads: 2
• Timer Stream Threads: 2
• Core Stream Commit Interval: 5 seconds
• Timer Stream Commit Interval: 30 seconds
• Core Memtable Size: 64MB
• Timer Memtable Size: 32MB
• Core State Store Cache Size: 32MB
• Timer State Store Cache Size: 64MB
• Shared RocksDB Block Cache Size: 64MB
• Write Buffer Manager Size: 4GB

This also took ~10 minutes. Turns out that this ticket is blocked by #576

[coltmcnealy-lh]

We might want to open a tech-debt ticket to make this class smaller.

[coltmcnealy-lh]

I disabled metrics aggregation (MetricsUpdater#listen()) and then ran the benchmark again. Prior to the changes in this PR, the benchmark took 119 seconds. After the changes, running with larger-memory.config took 113 seconds.

I disabled the changes related to the WriteBufferManager and Block Cache size (no shared WBM or Block Cache). Keeping the changes to commit interval and state store cache kept the small performance improvement (113 seconds). My thoughts going into this PR were confirmed:

• The changes to the WriteBufferManager and Block Cache are intended to reduce the risk of OOM errors.
• Using a larger commit interval should result in better performance. This performance change should be more drastic when running in a replicated environment with network latency (I am using local dev right now).

I think this PR is ready to review on its own. We should define follow-up issues to:

• Enable jemalloc on our docker image
• Determine recommended settings for the various cache sizes based on a given amount of memory.

My profiling has confirmed that this ticket accomplishes what it needed to do—power users can mostly limit the total on-heap and off-heap memory by tuning configurations, and we also improved performance slightly in the case where more resources are provided to the server.

The only danger of this PR is that the new default configurations result in poorer performance than the old default configurations, because for certain types of memory the old configurations did not specify a limit (which is dangerous as it makes OOM more easy to occur)

[coltmcnealy-lh]

@mjsax this appears to be the only way to "inject" configuration into the RockDBConfigSetter implementation. Would it be worth a KIP to add a method:

KafkaStreams#setRocksDBConfigurator(RocksDBConfigSetter otterSetter)

The method should throw IllegalStateException if it is called after KafkaStreams#start(), or if the rocksdb.config.setter parameter is set on the properties.

[mjsax]

Don't think it's the only way to inject configs.

There is RocksConfigSetter.setConfig(..., config) taking the Kafka Streams config object and third parameter. When you create StreamsConfig you pass in Map<?, ?> props and all entries will be contained in config.

Thus, you can take LHServerConfig and add all of it to props before creating StreamsConfig and should have access to all of it in setConfig(...)

[coltmcnealy-lh]

@mjsax Thanks so much! This works great. Much better than submitting a KIP to the Apache Kafka Debate Club 😄

CC @eduwercamacaro let's refactor this tomorrow.