-
Notifications
You must be signed in to change notification settings - Fork 0
Observability
CCX delivers the observability stack as a separate deployment available as a Helm chart here - https://github.com/severalnines/observability/.
This page describes how to set up observability. The observability solution for CCX relies heavily on VictoriaMetrics/VM-alerts (or Prometheus/AlertManager) with Grafana for visualisation. We recommend using VictoriaMetrics and hence Prometheus is not covered in any more detail in this document.
Moreover, the datastores are also visualized in the ClusterControl UI.
This stack uses:
- Victoria metrics - Prometheus compatibile observability for metrics and alerting (vmsingle/vmalert) - https://docs.victoriametrics.com/Single-server-VictoriaMetrics.html
- Alertmanager - https://prometheus.io/docs/alerting/latest/alertmanager/
- Grafana - https://grafana.com/docs/
- kube-state-metrics - https://github.com/kubernetes/kube-state-metrics
- kubernetes cluster
- helm - https://helm.sh/docs/intro/install/
- ccx-monitoring helm chart - https://github.com/severalnines/observability/
It's preferred to have a dedicated namespace for this stack. Create namespace:
kubectl create ns monitoring
kubectl config set-context --current --namespace=monitoring
helm repo add s9s https://severalnines.github.io/helm-charts/
helm repo update
helm pull s9s/ccx-monitoring --untar
Update your helm dependencies by running:
helm dependencies update
It is very important to edit values file and set variables like cluster name, ingress host, etc. Please make sure to go through values file and modify it as necessary.
helm show values ccx-monitoring > YOUR_VALUES.yaml
helm upgrade --install ccx-monitoring ccx-monitoring --values YOUR_VALUES.yaml --debug
Please note that this guide as some other documents and URLs assumes that you use ccx-monitoring release name.
Grafana dashboards and datasources are kept as a code in the repo.
kubectl delete -k ccx-monitoring/dashboards
kubectl create -k ccx-monitoring/dashboards
Additional dashboards can be placed as .json files in folders inside dashboards directory.
Grafana dashboards and datasources are kept as a code in the repo. Navigate to datasources directory, edit grafana_datasource.yaml and apply using:
kubectl apply -k ccx-monitoring/datasources
To retrieve the admin password (with username admin):
kubectl get secret ccx-monitoring-grafana -o jsonpath="{.data.admin-password}" | base64 --decode ; echo
To uninstall the monitoring, run:
helm uninstall ccx-monitoring
To integrate with slack, please see example in values.yaml file. See https://prometheus.io/docs/alerting/latest/configuration/ - for various integration and configuration options.