MONITORING.md

Monitoring

Optional components related monitoring the cluster.

Recommended setup:

pulumi config set beszel:enabled true
pulumi up

# copy key from UI
pulumi config set beszel:hubKey <KEY>
pulumi up

# add hosts using UI

Beszel

Homepage	https://beszel.dev/
Endpoints	https://beszel.<tsnet>.ts.net/
	https://beszel.<tsnet>.ts.net/_/

A lightweight alternative to Prometheus.

First deploy Beszel hub with:

pulumi config set beszel:enabled true
pulumi up

Once the hub is deployed, go to beszel.<tsnet>.ts.net endpoint and create an admin account.

To deploy agents you need to find the generated public key. Click Add system, then copy the Public key field. Close the popup and do not add any systems yet.

# replace <KEY> with the copied value "ssh-ed25519 ..."
pulumi config set beszel:hubKey <KEY>
pulumi up

Make sure to allow traffic to agents on port 45876:

firewall-cmd --permanent --add-port=45876/tcp

Once the agents are deployed, you need to manually add them in the UI of Beszel. Click Add system, select docker, then enter the hostname in the Name field and Tailscale IP in Host/IP

You can find the IP address of your node using one of two ways:

# List all hosts and IPs
tailscale status

# List only nodes added to cluster
kubectl get nodes -o json | jq -r '.items[] | .metadata.name + " - " + .metadata.annotations["flannel.alpha.coreos.com/public-ip"]'

Prometheus

Homepage	https://prometheus.io/
Helm chart	https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack
Endpoints	https://grafana.<tsnet>.ts.net/
	https://prometheus.<tsnet>.ts.net/
	https://alertmanager.<tsnet>.ts.net/

Prometheus provides much more detailed monitoring of the cluster. Many tools (like Headlamp) integrate with it to show metrics for Kubernetes resources.

Enabling it will increase traffic between nodes. Expect over 1GB of data saved to storage per day, even with just a few nodes.

pulumi config set prometheus:enabled true

# (optional) Override grafana "admin" password
pulumi config set prometheus:grafana-password <password> --secret

pulumi up

Grafana dashboards

Once Prometheus is installed, additional metrics and Grafana dashboards can be enabled for applications that support it.

The Grafana provider requires url and auth to be set in order to connect to the provisioned Grafana instance.

# Enable additional metrics and dashboards
# IMPORTANT: only enable once Prometheus has been installed.
pulumi config set prometheus:enableComponentMonitoring true

# Use your Tailnet domain name
pulumi config set grafana:url https://grafana.<tsnet>.ts.net/

# Basic auth user:password to Grafana frontend
pulumi config set grafana:auth admin:admin --secret

pulumi up

Uninstall

# Remove application monitoring before uninstalling Prometheus
pulumi config set prometheus:enableComponentMonitoring false

pulumi up

# Remove Prometheus
pulumi config set prometheus:enabled false

pulumi up

CRDs need to be removed manually, more info at https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack#uninstall-helm-chart