metrics: Add podman deployment

metrics/README.md

@@ -5,32 +5,51 @@ Our CI system regularly [builds](https://github.com/cockpit-project/bots/blob/ma
 These kubernetes resources deploy [Prometheus](https://prometheus.io/) to
 regularly read these metrics and store them in a database, and [Grafana](https://grafana.com/) to visualize these metrics as graphs.
 
-## Deployment
+## Deployment to Kubernetes
 
  - For the first-ever installation, create a persistent volume claim to store
    the Prometheus database (as that is somewhat precious):
 
-       oc create -f prometheus-claim.yaml
+       kubectl create -f prometheus-claim.yaml
 
    Once this gets bound, it is ready to use. If that does not happen
    automatically, file a support ticket like [#341](https://pagure.io/centos-infra/issue/341).
 
  - Whenever the YAML resources or the dashboards change, this script cleans up all old resources and re-deploys everything:
 
-       metrics/deploy.sh
+       metrics/deploy-k8s.sh
 
    After that, Grafana should be available at https://grafana-cockpit.apps.ocp.cloud.ci.centos.org and show the Cockpit CI dashboard at https://grafana-cockpit.apps.ocp.cloud.ci.centos.org/d/ci/cockpit-ci
 
 
+## Local deployment to podman
+
+For development, you can also deploy everything into podman:
+
+    metrics/deploy-podman.sh
+
+Note that if you have `cockpit.socket` running, this will conflict on port
+9090, so stop that first.
+
+This will remove a previous deployment and volumes, except for the
+`prometheus-data` one. If you want to start from scratch, clean that up with
+
+    podman volume rm prometheus-data
+
+Then you can access Prometheus on [localhost:9090](http://localhost:9090) and
+Grafana on [localhost:3000](http://localhost:3000). You have to log into
+Grafana as "admin:foobar" and go to Menu → Dashboards to see the deployed
+boards.
+
 ## Dashboard maintenance
 
-You can log into Grafana with "Sign in" from the left menu bar at the bottom, as user `admin`. The password is in the [internal CI secrets repository](https://gitlab.cee.redhat.com/front-door-ci-wranglers/ci-secrets/-/blob/master/metrics/grafana-admin).
+You can log into Grafana with "Sign in" from the left menu bar at the bottom, as user `admin`. The password is in the [internal CI secrets repository](https://gitlab.cee.redhat.com/front-door-ci-wranglers/ci-secrets/-/blob/master/metrics/grafana-admin) for k8s deployment, or "foobar" for local podman deployment.
 
 The metrics are meant to implement and measure our [Service Level objectives](https://github.com/cockpit-project/cockpit/wiki/DevelopmentPrinciples#our-testsci-error-budget). They are not complete yet.
 
 Whenever you change the dashboard, use the "Dashboard settings" button (cog
 icon at the top right) → JSON model, copy&paste it to
-[cockpit.ci-json](./cockpit-ci.json), and send a pull request to update it.
+[cockpit.ci-json](./dashboards/cockpit-ci.json), and send a pull request to update it.
 
 ## CI weather
 

metrics/deploy.sh → metrics/deploy-k8s.sh

@@ -8,7 +8,7 @@ kubectl delete configmap/grafana-dashboards || true
 kubectl delete -f $MYDIR/metrics.yaml || true
 
 filearg=""
-for f in $MYDIR/*.json; do
+for f in $MYDIR/dashboards/*; do
     filearg="$filearg --from-file $f"
 done
 kubectl create configmap grafana-dashboards $filearg

metrics/deploy-podman.sh

@@ -0,0 +1,41 @@
+#!/bin/sh
+# (Re)deploy Prometheus/Grafana to local podman, for development
+# Note: This keeps the prometheus-data volume
+set -eux
+MYDIR=$(realpath -m "$0"/..)
+
+# clean up old deployment
+podman kube down "${MYDIR}/metrics.yaml"
+for filter in grafana prometheus-config; do
+    podman volume ls -q --filter "name=$filter" | xargs --no-run-if-empty podman volume rm
+done
+
+# "foobar" password secret for Grafana "admin" user
+cat << EOF | podman kube play -
+---
+apiVersion: v1
+kind: Secret
+metadata:
+  name: metrics-secrets
+data:
+  grafana-admin: Zm9vYmFy
+EOF
+
+# adjust the k8s deployment to work for podman play kube: grafana-dashboards ConfigMap gets built dynamically in
+# metrics/deploy-k8s.sh, but it is much more flexible and ergonomic to just mount the dashboards directory
+patch -o- "$MYDIR/metrics.yaml" - <<EOF | podman kube play -
+--- metrics/metrics.yaml
++++ metrics/metrics.yaml
+@@ -75,9 +73,8 @@ spec:
+           configMap:
+             name: grafana-provisioning-dashboards
+         - name: grafana-dashboards
+-          configMap:
+-            # this is not defined here, but gets built from *.json files in ./deploy.sh
+-            name: grafana-dashboards
++          hostPath:
++            path: metrics/dashboards
+
+ ---
+ kind: ConfigMap
+EOF