From 7ffb423ce8cdf718d0a650771f32b1fce6f219d8 Mon Sep 17 00:00:00 2001 From: sj-williams Date: Tue, 19 Dec 2023 17:36:18 +0000 Subject: [PATCH] update dashboard duplication troubleshooting --- .../source/grafana-dashboards.html.md.erb | 33 +++++++++++++++++-- 1 file changed, 31 insertions(+), 2 deletions(-) diff --git a/runbooks/source/grafana-dashboards.html.md.erb b/runbooks/source/grafana-dashboards.html.md.erb index a2c5b4d6..975678b6 100644 --- a/runbooks/source/grafana-dashboards.html.md.erb +++ b/runbooks/source/grafana-dashboards.html.md.erb @@ -1,7 +1,7 @@ --- title: Grafana Dashboards weight: 9106 -last_reviewed_on: 2023-11-20 +last_reviewed_on: 2023-12-19 review_in: 3 months --- @@ -46,5 +46,34 @@ You'll see an error similar to: ```bash t=2021-12-03T13:37:35+0000 lvl=eror msg="failed to load dashboard from " logger=provisioning.dashboard type=file name=sidecarProvider file=/tmp/dashboards/.json error="invalid character 'c' looking for beginning of value" ``` +To help in identifying the dashboards, you can exec into the Grafana pod as follows: -Identify that dashboard and fix the error in question. +``` +kubectl exec --stdin --tty prometheus-operator-grafana-xxxxxx-yyyyy --container grafana-sc-dashboard -- sh +``` + +Then, cd into the `/tmp/dashboards` directory, and execute the `find` command as below: + +``` +cd /tmp/dashboards/ +for i in $(find ./ -type f); do tail $i; done|grep '"uid": '|cut -f2 -d':'|sort|uniq -c|grep -v ' 1 ' +``` +This will show output any duplicate UIDs and the number of occurrences of each. + +ie: + +``` +2 "[duplicate-dashboard-uid]", +... +``` + +Now you can grep the dashboard `json` files within the directory to identify which dashboards contain the duplicate UID with: + +``` +grep -Rnw . -e "[duplicate-dashboard-uid]" + +./my-test-dashboard.json: "uid": "duplicate-dashboard-uid", +./my-test-dashboard-2.json: "uid": "duplicate-dashboard-uid", +``` + +Identify that dashboard and fix the error in question, depending on where the dashboard config itself is created you may need to identify the user who created the dashboard and ask them to fix it.