metrics: Add podman deployment

This makes working on metrics easy for development and experimentation.

metrics/README.md

@@ -22,9 +22,28 @@ regularly read these metrics and store them in a database, and [Grafana](https:/
    After that, Grafana should be available at https://grafana-cockpit.apps.ocp.cloud.ci.centos.org and show the Cockpit CI dashboard at https://grafana-cockpit.apps.ocp.cloud.ci.centos.org/d/ci/cockpit-ci
 
 
+## Local deployment to podman
+
+For development, you can also deploy everything into podman:
+
+    metrics/deploy-podman.sh
+
+Note that if you have `cockpit.socket` running, this will conflict on port
+9090, so stop that first.
+
+This will remove a previous deployment and volumes, except for the
+`prometheus-data` one. If you want to start from scratch, clean that up with
+
+    podman volume rm prometheus-data
+
+Then you can access Prometheus on [localhost:9090](http://localhost:9090) and
+Grafana on [localhost:3000](http://localhost:3000). You have to log into
+Grafana as "admin:foobar" and go to Menu → Dashboards to see the deployed
+boards.
+
 ## Dashboard maintenance
 
-You can log into Grafana with "Sign in" from the left menu bar at the bottom, as user `admin`. The password is in the [internal CI secrets repository](https://gitlab.cee.redhat.com/front-door-ci-wranglers/ci-secrets/-/blob/master/metrics/grafana-admin).
+You can log into Grafana with "Sign in" from the left menu bar at the bottom, as user `admin`. The password is in the [internal CI secrets repository](https://gitlab.cee.redhat.com/front-door-ci-wranglers/ci-secrets/-/blob/master/metrics/grafana-admin) for k8s deployment, or "foobar" for local podman deployment.
 
 The metrics are meant to implement and measure our [Service Level objectives](https://github.com/cockpit-project/cockpit/wiki/DevelopmentPrinciples#our-testsci-error-budget). They are not complete yet.
 

metrics/deploy-podman.sh

@@ -0,0 +1,45 @@
+#!/bin/sh
+# (Re)deploy Prometheus/Grafana to local podman, for development
+# Note: This keeps the prometheus-data volume
+set -eux
+MYDIR=$(realpath -m "$0"/..)
+
+# clean up old deployment
+podman kube down "${MYDIR}/metrics.yaml"
+for filter in grafana prometheus-config; do
+    podman volume ls -q --filter "name=$filter" | xargs --no-run-if-empty podman volume rm
+done
+
+# adjust the k8s deployment to work for podman play kube:
+#  - grafana-dashboards ConfigMap gets built dynamically in metrics/deploy-k8s.sh, but
+#    it is much more flexible and ergonomic to just mount the dashboards directory
+#  - replace grafana-admin secret with static "foobar" admin password
+patch -o- "$MYDIR/metrics.yaml" - <<EOF | podman kube play -
+--- metrics/metrics.yaml
++++ metrics/metrics.yaml
+@@ -35,10 +35,8 @@ spec:
+         - name: GF_PATHS_PROVISIONING
+           value: /etc/grafana/provisioning
+         - name: GF_SECURITY_ADMIN_PASSWORD
+-          valueFrom:
+-            secretKeyRef:
+-              name: metrics-secrets
+-              key: grafana-admin
++          value: foobar
++
+         ports:
+         - containerPort: 3000
+           protocol: TCP
+@@ -75,9 +73,8 @@ spec:
+           configMap:
+             name: grafana-provisioning-dashboards
+         - name: grafana-dashboards
+-          configMap:
+-            # this is not defined here, but gets built from *.json files in ./deploy.sh
+-            name: grafana-dashboards
++          hostPath:
++            path: metrics/dashboards
+
+ ---
+ kind: ConfigMap
+EOF