使用 Prometheus 监控 k3s 集群

[Bpazy]

k3s 集群内置了 cadvisor，所以我们可以直接利用，核心是 prometheus 的 scrape_configs 中 kubernetes_sd_configs 相关配置:

    scrape_configs:
      - job_name: 'cadvisor'
        scheme: https
        api_server: https://exmaple.com:6443
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
        - role: node
        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

注意这里的 ca_file, bearer_token_file，这两个配置的值就这么写，prometheus 容器里的这个路径就是正确的，不需要再去做一些 mount。

为了能访问通 k3s，我们还需要创建 k8s role，否则看 prometheus 的日志就会发现异常:

level=error ts=2024-04-19T119:15:17.102Z caller=klog.go:94 component=k8s_client_runtime func=ErrorDepth msg="/app/discovery/kubernetes/kubernetes.go:335: Failed to list *v1.Node: nodes is forbidden: User \"system:serviceaccount:kube-mon:default\" cannot list resource \"nodes\" in API group \"\" at the cluster scope"

创建 role 的 yaml:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups:
  - ""
  resources:
  - nodes
  - services
  - endpoints
  - pods
  - nodes/proxy
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - "extensions"
  resources:
    - ingresses
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - configmaps
  - nodes/metrics
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: default

然后在 deployment.spec.template.spec.serviceAccountName 中指定上面创建的 role: prometheus。

其他配置就以自己的为准即可。

[Bpazy]

Grafana 可视化

在成功收集 k3s 集群的信息后，我们还需要可视化这些信息用于分析。

可以利用 https://grafana.com/grafana/dashboards/15282-k8s-rke-cluster-monitoring/ 这个已有的 Grafana Dashboard，效果如下: