Update docs #2213
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,60 @@ | ||
| ============================================= | ||
| ScyllaDB Cloud Monitoring Datadog Integration | ||
| ============================================= | ||
|
|
||
| For security reasons, the ScyllaDB cloud does not have direct access to the Prometheus server. | ||
| To allow external server scrapping, you will need to enable the Prometheus proxy. | ||
| The Datadog agent reads from the proxy, which reads from the Promethues server. | ||
|
|
||
| 1. Installing and configuring the Datadog Agent. | ||
| 2. Add Datadog recording rules. | ||
| 3. Loading Scylla dashboard to Datadog. | ||
| 4. Optionally load Monitor (Alerts). | ||
|
|
||
| Scylla Monitoring Datadog Integration Overview | ||
| ============================================== | ||
| A typical ScyllaDB cluster generates thousands of metrics, sometimes even tens of thousands. | ||
| The sheer number of metrics is too much for Datadog. | ||
|
|
||
| Instead of letting the Datadog agent scrap all metrics, the monitoring stack marks a small subset of metrics with a label and lets the Datadog agent scrap only those. | ||
|
|
||
| Install And configure the Datadog Agent | ||
| ======================================= | ||
|
|
||
| Start by following `Installation <https://docs.datadoghq.com/agent/>`_ guide. The datadog agent should run on a machine that can reach the Prometheus Proxy server. | ||
|
There was a problem hiding this comment. Its not clear whre to install the agent. A small digram would help. There was a problem hiding this comment. |
||
|
|
||
| Once the Datadog agent is working, download the configuration file and place it under /etc/datadog-agent/conf.d/prometheus.d/conf.yaml | ||
|
|
||
| Download the configuration file :download:`conf.yaml <cloud-conf.yaml>` move it to: /etc/datadog-agent/conf.d/prometheus.d/conf.yaml | ||
|
|
||
|
|
||
| Edit the file. You must replace the cluster id (CLUSTER_ID) and the token (TOKEN). | ||
|
|
||
| Post configuration | ||
| ^^^^^^^^^^^^^^^^^^ | ||
| Restart the agent based on your installation. Scylla metrics should be visible in Datadog. | ||
|
|
||
|
|
||
| .. note:: By default, Datadog will not scrap per-shard metrics. To enable per-shard metrics, edit the conf.yaml file and replace dd=~"1" with dd=~"1|2" | ||
|
|
||
| Upload the Dashboard | ||
| ==================== | ||
| Download the dashboard file :download:`dashboard.json <dashboard.json>`. | ||
| Create a new dashboard in Datadog and import the json file you downloaded. | ||
|
|
||
| Using the Dashboard | ||
| =================== | ||
| We created a Datadog dashboard that resembles the Grafana dashboards. | ||
|
|
||
| .. image:: datadog.png | ||
|
|
||
| The dashboard contains some specific filtering and perspectives: | ||
| First, you can choose between shard, instance, dc, or cluster view. | ||
| This will aggregate the metrics in the graphs accordingly. | ||
| Second, you can filter to see specific shards, nodes, or DCs. | ||
|
|
||
| .. note:: Pay attention that some of the combinations are conflicting. For example, you cannot filter by DC when looking at a cluster view. If no data displayed, remove the filters first. | ||
|
|
||
| Adding Monitor | ||
| ============== | ||
| Alerts in Datadog called Monitor. Download the monitor file :download:`monitor.json <monitor.json>`. Go to the Monitor section in datadog and import the json. | ||
|
There was a problem hiding this comment. |
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"For security reasons, the ScyllaDB cloud does not have direct access to the Prometheus server." This can be removed; it does not add useful information. We might move away from Prometheus in the future.