Add tuning documentation #9978
Open
Add tuning documentation #9978
+49
−0
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wilfriedroset
force-pushed
the
tuning-documentation
branch
from
150c581
to
239498c
Compare
|
ping @56quarters @bboreham for the review as you were helping on the proposal 🙏🏾 |
wilfriedroset
force-pushed
the
tuning-documentation
branch
from
239498c
to
335e33e
Compare
tacole02
approved these changes
Nov 22, 2024
| @@ -0,0 +1,47 @@ | |||
| --- | |||
| description: Learn how to tune Grafan Mimir according to your use cases. | |||
There was a problem hiding this comment.
Suggested change
| description: Learn how to tune Grafan Mimir according to your use cases. | |
| description: Tune Grafana Mimir according to your use cases. |
|
|
||
| # Tune Grafana Mimir according to your use cases | ||
|
|
||
| Grafana Mimir comes with sensible default settings. Those settings are a good place to start for most use cases. |
There was a problem hiding this comment.
Suggested change
| Grafana Mimir comes with sensible default settings. Those settings are a good place to start for most use cases. | |
| For most use cases, you can use the default settings that come with Mimir. |
There was a problem hiding this comment.
We refer to it as just "Mimir" throughout the documentation, so I think it's okay to do that here, too.
| # Tune Grafana Mimir according to your use cases | ||
|
|
||
| Grafana Mimir comes with sensible default settings. Those settings are a good place to start for most use cases. | ||
| However, for some use cases Grafana Mimir requires appropriate tuning to reach optimal performance. This page aims to centralize those known tuning. |
There was a problem hiding this comment.
Suggested change
| However, for some use cases Grafana Mimir requires appropriate tuning to reach optimal performance. This page aims to centralize those known tuning. | |
| However, sometimes you need to tune Mimir to reach optimal performance. Use the following guidance when tuning settings in Mimir.``` |
| ## Heavy multi-tenancy | ||
|
|
||
| For each tenant, Grafana Mimir opens and maintains a TSDB in memory. With a significant number of tenants the memory overhead might come prohibitive. | ||
| To reduce the associated overhead, users might consider: |
There was a problem hiding this comment.
Suggested change
| To reduce the associated overhead, users might consider: | |
| To reduce the associated overhead, consider the following: |
There was a problem hiding this comment.
We like to address users directly in the second person.
|
|
||
| ## Heavy multi-tenancy | ||
|
|
||
| For each tenant, Grafana Mimir opens and maintains a TSDB in memory. With a significant number of tenants the memory overhead might come prohibitive. |
There was a problem hiding this comment.
Suggested change
| For each tenant, Grafana Mimir opens and maintains a TSDB in memory. With a significant number of tenants the memory overhead might come prohibitive. | |
| For each tenant, Mimir opens and maintains a TSDB in memory. If you have a significant number of tenants, the memory overhead might become prohibitive. |
| Grafana Mimir relies on Memcached for its caches. Memcached relies, by default only on the memory. | ||
| Similarly to the work of the Ops team behind Grafana Loki, one could enable the `extstore` feature of Memcached. | ||
|
|
||
| See: [how we scaled Grafana Cloud Logs' Memcached cluster to 50TB and improved reliability](https://grafana.com/blog/2023/08/23/how-we-scaled-grafana-cloud-logs-memcached-cluster-to-50tb-and-improved-reliability/) |
There was a problem hiding this comment.
Suggested change
| See: [how we scaled Grafana Cloud Logs' Memcached cluster to 50TB and improved reliability](https://grafana.com/blog/2023/08/23/how-we-scaled-grafana-cloud-logs-memcached-cluster-to-50tb-and-improved-reliability/) | |
| Refer to [how we scaled Grafana Cloud Logs' Memcached cluster to 50TB and improved reliability](https://grafana.com/blog/2023/08/23/how-we-scaled-grafana-cloud-logs-memcached-cluster-to-50tb-and-improved-reliability/). |
|
|
||
| ## Periodic latency spikes when cutting blocks | ||
|
|
||
| Depending on the workload, users might witness latency spikes when Grafana Mimir cuts blocks. |
There was a problem hiding this comment.
Suggested change
| Depending on the workload, users might witness latency spikes when Grafana Mimir cuts blocks. | |
| Depending on the workload, you might witness latency spikes when Mimir cuts blocks. |
| ## Periodic latency spikes when cutting blocks | ||
|
|
||
| Depending on the workload, users might witness latency spikes when Grafana Mimir cuts blocks. | ||
| To reduce the impacts of this behavior, users might consider: |
There was a problem hiding this comment.
Suggested change
| To reduce the impacts of this behavior, users might consider: | |
| To reduce the impact of this behavior, consider the following: |
| Depending on the workload, users might witness latency spikes when Grafana Mimir cuts blocks. | ||
| To reduce the impacts of this behavior, users might consider: | ||
|
|
||
| - Upgrade to `2.15+`, See: <https://github.com/grafana/mimir/commit/03f2f06e1247e997a0246d72f5c2c1fd9bd386df> |
There was a problem hiding this comment.
Suggested change
| - Upgrade to `2.15+`, See: <https://github.com/grafana/mimir/commit/03f2f06e1247e997a0246d72f5c2c1fd9bd386df> | |
| - Upgrade to `2.15+`. Refer to <https://github.com/grafana/mimir/commit/03f2f06e1247e997a0246d72f5c2c1fd9bd386df>. |
| To reduce the impacts of this behavior, users might consider: | ||
|
|
||
| - Upgrade to `2.15+`, See: <https://github.com/grafana/mimir/commit/03f2f06e1247e997a0246d72f5c2c1fd9bd386df> | ||
| - Reduce `-blocks-storage.tsdb.block-ranges-period`, default `2h`, For example try `1h` |
There was a problem hiding this comment.
Suggested change
| - Reduce `-blocks-storage.tsdb.block-ranges-period`, default `2h`, For example try `1h` | |
| - Reduce `-blocks-storage.tsdb.block-ranges-period`, default `2h`. For example. try `1h`. |
This commit is a followup of the PR grafana#9149 where we discussed the idea of centralizing the known tuning associated with their use cases. Signed-off-by: Wilfried Roset <[email protected]>
wilfriedroset
force-pushed
the
tuning-documentation
branch
from
335e33e
to
5a1294d
Compare
|
thank you @tacole02 for your review I've taken into account your remarks. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does
This commit is a followup of the PR #9149 where we discussed the idea of centralizing the known tuning associated with their use cases.
Which issue(s) this PR fixes or relates to
Relates to #9149
Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX].about-versioning.mdupdated with experimental features.