From 380b62e898869660d87d0acf3f22b67df4afd0c2 Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Fri, 9 Sep 2022 15:22:23 +0200 Subject: [PATCH 01/30] added small tutorial on how to bring your custom kube-prometheus-stack --- prometheus/README.md | 103 +++++++++++++++++++++++++++++++++++++++++ prometheus/values.yaml | 57 +++++++++++++++++++++++ 2 files changed, 160 insertions(+) create mode 100644 prometheus/README.md create mode 100644 prometheus/values.yaml diff --git a/prometheus/README.md b/prometheus/README.md new file mode 100644 index 00000000..7475cdd4 --- /dev/null +++ b/prometheus/README.md @@ -0,0 +1,103 @@ +# Install a custom kube-prometheus-stack in Kyma + +## Overview + +The Kyma monitoring stack often brings limited configuration options in contrast to the upstream [`kube-prometheus-stack`](https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack) chart. Modifications might be resetted at next upgrade cycle. + +An alternative can be a parallel installation of the upstream chart offering all customization options. This tutorial outlines how to achieve such installation in co-existent to the Kyma monitoring stack. + +## Prerequisites + +- Kyma as the target deployment environment. +- kubectl > 1.22.x +- helm 3.x + +## Installation + +### Preparation + +1. Export your Namespace as a variable. Replace the `{namespace}` placeholder in the following command and run it: + + ```bash + export KYMA_EXAMPLE_NS="{namespace}" + ``` + +1. Export the Helm release name which you want to use. It can be any name,, be aware that all resources in the cluster will be prefixed with that name. Replace the `{release-name}` placeholder in the following command and run it: + ```bash + export HELM_RELEASE_NAME="{release-name}" + ``` + +2. Update your helm installation with the required helm repository: + + ```bash + helm repo add prometheus-community https://prometheus-community.github.io/helm-charts + helm repo update + ``` + +### Install the kube-prometheus-stack + +1. Run the Helm upgrade command which will install the chart if not present yet. Change the grafana admin password (at the end of the command) to some value of your choice + ```bash + helm upgrade --install -n ${KYMA_EXAMPLE_NS} ${HELM_RELEASE_NAME} prometheus-community/kube-prometheus-stack -f values.yaml --set grafana.adminPassword=myPwd + ``` + +Hereby, use the [values.yaml](./values.yaml) provided with this tutorial which contains customized settings derivating from the default settings. + +### Verify the installation + +1. You should see several pods coming up in the namespace, especially prometheus and alertmanager. Assure that all pods are ending in a "Running" state. +2. Browse the prometheus dashboard and verify that all "Status->Targets" are healthy. Following command will expose the dashboard on `http://localhost:9090` + ```bash + kubectl -n ${KYMA_EXAMPLE_NS} port-forward svc/${HELM_RELEASE_NAME}-kube-prometheus-stack-prometheus 9090 + ``` +2. Browse the grafana dashboard and verify that the dashboards are showing data. Following command will expose the dashboard on `http://localhost:3000` + ```bash + kubectl -n ${KYMA_EXAMPLE_NS} port-forward svc/${HELM_RELEASE_NAME}-grafana 3000:80 + ``` + +### Deploy a custom workload and scrape it + +1. Deploy a custom application which exposes metrics: + ```bash + kubectl create namespace myapp + kubectl label namespace myapp istio-injection=enabled + kubectl apply -n myapp -f https://raw.githubusercontent.com/kyma-project/examples/main/monitoring-custom-metrics/deployment/deployment.yaml + ``` + +2. Configure prometheus to scrape the new target by defining a ServiceMonitor: + ```bash + kubectl apply -n myapp -f - < Date: Fri, 9 Sep 2022 15:37:13 +0200 Subject: [PATCH 02/30] fixes --- CODEOWNERS | 6 +++++ monitoring-custom-metrics/README.md | 8 ++++-- prometheus/README.md | 42 +++++------------------------ 3 files changed, 18 insertions(+), 38 deletions(-) diff --git a/CODEOWNERS b/CODEOWNERS index ed995e22..0b30a965 100644 --- a/CODEOWNERS +++ b/CODEOWNERS @@ -19,6 +19,12 @@ # Custom function runtime image example /custom-serverless-runtime-image/ @m00g3n @pPrecel @dbadura @kwiatekus @moelsayed @cortey +/prometheus/ @a-thaler @rakesh-garimella @skhalash @chrkl @lindnerby @dennis-ge @lilitgh + +/monitoring-custom-metrics/ @a-thaler @rakesh-garimella @skhalash @chrkl @lindnerby @dennis-ge @lilitgh + +/monitoring-custom-alerts/ @a-thaler @rakesh-garimella @skhalash @chrkl @lindnerby @dennis-ge @lilitgh + # Scaling functions to zero with KEDA /scale-to-zero-with-kedam @m00g3n @pPrecel @dbadura @kwiatekus @moelsayed @cortey diff --git a/monitoring-custom-metrics/README.md b/monitoring-custom-metrics/README.md index 3c49b55c..ed00de99 100644 --- a/monitoring-custom-metrics/README.md +++ b/monitoring-custom-metrics/README.md @@ -21,13 +21,17 @@ This example shows how to expose custom metrics to Prometheus with a Golang serv export KYMA_EXAMPLE_NS="{namespace}" ``` -2. Deploy the service: +2. Ensure that your namespace is having Istio sidecar injection enabled, the example will assume that the metrics are exposed ina strict mTLS mode: + ```` + kubectl label namespace ${KYMA_EXAMPLE_NS} istio-injection=enabled + +3. Deploy the service: ```bash kubectl apply -f deployment/deployment.yaml -n $KYMA_EXAMPLE_NS ``` -3. Deploy the ServiceMonitor: +4. Deploy the ServiceMonitor: ```bash kubectl apply -f deployment/service-monitor.yaml diff --git a/prometheus/README.md b/prometheus/README.md index 7475cdd4..75e3ef10 100644 --- a/prometheus/README.md +++ b/prometheus/README.md @@ -38,10 +38,13 @@ An alternative can be a parallel installation of the upstream chart offering all 1. Run the Helm upgrade command which will install the chart if not present yet. Change the grafana admin password (at the end of the command) to some value of your choice ```bash - helm upgrade --install -n ${KYMA_EXAMPLE_NS} ${HELM_RELEASE_NAME} prometheus-community/kube-prometheus-stack -f values.yaml --set grafana.adminPassword=myPwd + helm upgrade --install -n ${KYMA_EXAMPLE_NS} ${HELM_RELEASE_NAME} prometheus-community/kube-prometheus-stack -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/values.yaml --set grafana.adminPassword=myPwd ``` -Hereby, use the [values.yaml](./values.yaml) provided with this tutorial which contains customized settings derivating from the default settings. +Hereby, use the [values.yaml](./values.yaml) provided with this tutorial which contains customized settings deviating from the default settings, or create your own one. +The provided values.yaml covers the following adjustments: +- Support parallel operation to a Kyma monitoring stack +- Scraping of Istio strict mTLS workloads ### Verify the installation @@ -57,36 +60,7 @@ Hereby, use the [values.yaml](./values.yaml) provided with this tutorial which c ### Deploy a custom workload and scrape it -1. Deploy a custom application which exposes metrics: - ```bash - kubectl create namespace myapp - kubectl label namespace myapp istio-injection=enabled - kubectl apply -n myapp -f https://raw.githubusercontent.com/kyma-project/examples/main/monitoring-custom-metrics/deployment/deployment.yaml - ``` - -2. Configure prometheus to scrape the new target by defining a ServiceMonitor: - ```bash - kubectl apply -n myapp -f - < Date: Fri, 9 Sep 2022 15:38:52 +0200 Subject: [PATCH 03/30] fix --- prometheus/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/prometheus/README.md b/prometheus/README.md index 75e3ef10..cfa83ae7 100644 --- a/prometheus/README.md +++ b/prometheus/README.md @@ -53,7 +53,7 @@ The provided values.yaml covers the following adjustments: ```bash kubectl -n ${KYMA_EXAMPLE_NS} port-forward svc/${HELM_RELEASE_NAME}-kube-prometheus-stack-prometheus 9090 ``` -2. Browse the grafana dashboard and verify that the dashboards are showing data. Following command will expose the dashboard on `http://localhost:3000` +2. Browse the grafana dashboard and verify that the dashboards are showing data. The user `admin` is pre-configured in the provided values.yaml, the password was provided in your helm install command. Following command will expose the dashboard on `http://localhost:3000`: ```bash kubectl -n ${KYMA_EXAMPLE_NS} port-forward svc/${HELM_RELEASE_NAME}-grafana 3000:80 ``` From 98ce84c965befa8499013d7dcc5deee368b879b3 Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Mon, 12 Sep 2022 10:55:47 +0200 Subject: [PATCH 04/30] update --- prometheus/README.md | 2 +- prometheus/values.yaml | 5 ----- 2 files changed, 1 insertion(+), 6 deletions(-) diff --git a/prometheus/README.md b/prometheus/README.md index cfa83ae7..44242061 100644 --- a/prometheus/README.md +++ b/prometheus/README.md @@ -53,7 +53,7 @@ The provided values.yaml covers the following adjustments: ```bash kubectl -n ${KYMA_EXAMPLE_NS} port-forward svc/${HELM_RELEASE_NAME}-kube-prometheus-stack-prometheus 9090 ``` -2. Browse the grafana dashboard and verify that the dashboards are showing data. The user `admin` is pre-configured in the provided values.yaml, the password was provided in your helm install command. Following command will expose the dashboard on `http://localhost:3000`: +2. Browse the grafana dashboard and verify that the dashboards are showing data. The user `admin` is pre-configured in the helm chart, the password was provided in your helm install command. Following command will expose the dashboard on `http://localhost:3000`: ```bash kubectl -n ${KYMA_EXAMPLE_NS} port-forward svc/${HELM_RELEASE_NAME}-grafana 3000:80 ``` diff --git a/prometheus/values.yaml b/prometheus/values.yaml index b35905e9..2b491acc 100644 --- a/prometheus/values.yaml +++ b/prometheus/values.yaml @@ -50,8 +50,3 @@ prometheus: volumeMounts: - mountPath: /etc/prometheus/secrets/istio.default/ name: istio-certs - -####### This block configures Grafana -grafana: - adminUser: admin - #adminPassword: strongpassword From d69eb893942c33d8629fb56ceb0d28a5a369a019 Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Mon, 12 Sep 2022 17:30:09 +0200 Subject: [PATCH 05/30] fix --- prometheus/values.yaml | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/prometheus/values.yaml b/prometheus/values.yaml index 2b491acc..415ff113 100644 --- a/prometheus/values.yaml +++ b/prometheus/values.yaml @@ -12,10 +12,16 @@ prometheus-node-exporter: port: 9101 targetPort: 9101 -####### This block is disabling ETCD as it is not available in the data plane +####### This block is disabling control plane components which are not available in the Gardener data plane kubeEtcd: enabled: false +kubeControllerManager: + enabled: false +kubeProxy: + enabled: false +kubeScheduler: + enabled: false ####### This block is needed to also use ServiceMonitors which are not deployed as part of the chart prometheus: From 673e74b8639da848ddef21613a67dc012ab3da2f Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Tue, 13 Sep 2022 13:42:28 +0200 Subject: [PATCH 06/30] fix --- prometheus/README.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/prometheus/README.md b/prometheus/README.md index 44242061..2eb957ab 100644 --- a/prometheus/README.md +++ b/prometheus/README.md @@ -6,11 +6,13 @@ The Kyma monitoring stack often brings limited configuration options in contrast An alternative can be a parallel installation of the upstream chart offering all customization options. This tutorial outlines how to achieve such installation in co-existent to the Kyma monitoring stack. +Be aware of that this tutorial describes a basic setup which is not to be used in production. Further configuration is usually required like optimizations in regards to the amount of data to scrape and the required resource footprint of the installation. Even a different setup might be required to achieve qualities like high availability, scalability or durable long-term storage. + ## Prerequisites - Kyma as the target deployment environment. -- kubectl > 1.22.x -- helm 3.x +- Kubectl > 1.22.x +- Helm 3.x ## Installation From 627da9b963cdf2d4b3a3a9f29e78ea3d3b5efa26 Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Thu, 15 Sep 2022 14:37:40 +0200 Subject: [PATCH 07/30] Apply suggestions from code review Co-authored-by: Nina Hingerl <76950046+NHingerl@users.noreply.github.com> --- monitoring-custom-metrics/README.md | 3 ++- prometheus/README.md | 28 ++++++++++++++-------------- prometheus/values.yaml | 16 ++++++++-------- 3 files changed, 24 insertions(+), 23 deletions(-) diff --git a/monitoring-custom-metrics/README.md b/monitoring-custom-metrics/README.md index ed00de99..fcc5de4c 100644 --- a/monitoring-custom-metrics/README.md +++ b/monitoring-custom-metrics/README.md @@ -21,7 +21,8 @@ This example shows how to expose custom metrics to Prometheus with a Golang serv export KYMA_EXAMPLE_NS="{namespace}" ``` -2. Ensure that your namespace is having Istio sidecar injection enabled, the example will assume that the metrics are exposed ina strict mTLS mode: +2. Ensure that your Namespace has Istio sidecar injection enabled. This example assumes that the metrics are exposed in a strict mTLS mode: + ```` kubectl label namespace ${KYMA_EXAMPLE_NS} istio-injection=enabled diff --git a/prometheus/README.md b/prometheus/README.md index 2eb957ab..68d6331a 100644 --- a/prometheus/README.md +++ b/prometheus/README.md @@ -2,9 +2,9 @@ ## Overview -The Kyma monitoring stack often brings limited configuration options in contrast to the upstream [`kube-prometheus-stack`](https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack) chart. Modifications might be resetted at next upgrade cycle. +The Kyma monitoring stack often brings limited configuration options in contrast to the upstream [`kube-prometheus-stack`](https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack) chart. Modifications might be reset at the next upgrade cycle. -An alternative can be a parallel installation of the upstream chart offering all customization options. This tutorial outlines how to achieve such installation in co-existent to the Kyma monitoring stack. +As an alternative, you can install the upstream chart with all customization options parallel. This tutorial outlines how to set up such installation in co-existence to the Kyma monitoring stack. Be aware of that this tutorial describes a basic setup which is not to be used in production. Further configuration is usually required like optimizations in regards to the amount of data to scrape and the required resource footprint of the installation. Even a different setup might be required to achieve qualities like high availability, scalability or durable long-term storage. @@ -24,12 +24,12 @@ Be aware of that this tutorial describes a basic setup which is not to be used i export KYMA_EXAMPLE_NS="{namespace}" ``` -1. Export the Helm release name which you want to use. It can be any name,, be aware that all resources in the cluster will be prefixed with that name. Replace the `{release-name}` placeholder in the following command and run it: +2. Export the Helm release name that you want to use. It can be any name, but be aware that all resources in the cluster will be prefixed with that name. Replace the `{release-name}` placeholder in the following command and run it: ```bash export HELM_RELEASE_NAME="{release-name}" ``` -2. Update your helm installation with the required helm repository: +3. Update your Helm installation with the required Helm repository: ```bash helm repo add prometheus-community https://prometheus-community.github.io/helm-charts @@ -38,37 +38,37 @@ Be aware of that this tutorial describes a basic setup which is not to be used i ### Install the kube-prometheus-stack -1. Run the Helm upgrade command which will install the chart if not present yet. Change the grafana admin password (at the end of the command) to some value of your choice +1. Run the Helm upgrade command, which installs the chart if it's not present yet. At the end of the command, change the Grafana admin password to some value of your choice. ```bash helm upgrade --install -n ${KYMA_EXAMPLE_NS} ${HELM_RELEASE_NAME} prometheus-community/kube-prometheus-stack -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/values.yaml --set grafana.adminPassword=myPwd ``` -Hereby, use the [values.yaml](./values.yaml) provided with this tutorial which contains customized settings deviating from the default settings, or create your own one. -The provided values.yaml covers the following adjustments: +You can use the [values.yaml](./values.yaml) provided with this tutorial, which contains customized settings deviating from the default settings, or create your own one. +The provided `values.yaml` covers the following adjustments: - Support parallel operation to a Kyma monitoring stack -- Scraping of Istio strict mTLS workloads +- Support the scraping of workload secured with Istio strict mTLS ### Verify the installation -1. You should see several pods coming up in the namespace, especially prometheus and alertmanager. Assure that all pods are ending in a "Running" state. -2. Browse the prometheus dashboard and verify that all "Status->Targets" are healthy. Following command will expose the dashboard on `http://localhost:9090` +1. You should see several Pods coming up in the Namespace, especially Prometheus and Alertmanager. Assure that all Pods have the "Running" state. +2. Browse the Prometheus dashboard and verify that all "Status->Targets" are healthy. The following command exposes the dashboard on `http://localhost:9090`: ```bash kubectl -n ${KYMA_EXAMPLE_NS} port-forward svc/${HELM_RELEASE_NAME}-kube-prometheus-stack-prometheus 9090 ``` -2. Browse the grafana dashboard and verify that the dashboards are showing data. The user `admin` is pre-configured in the helm chart, the password was provided in your helm install command. Following command will expose the dashboard on `http://localhost:3000`: +3. Browse the Grafana dashboard and verify that the dashboards are showing data. The user `admin` is pre-configured in the Helm chart; the password was provided in your `helm install` command. The following command exposes the dashboard on `http://localhost:3000`: ```bash kubectl -n ${KYMA_EXAMPLE_NS} port-forward svc/${HELM_RELEASE_NAME}-grafana 3000:80 ``` ### Deploy a custom workload and scrape it -1. Follow the tutorial [monitoring-custom-metrics](./../monitoring-custom-metrics/) but use the steps above for verifying that the metrics are collected. +1. Follow the tutorial [monitoring-custom-metrics](./../monitoring-custom-metrics/), but use the steps above to verify that the metrics are collected. ### Cleanup -Run the following commands to completely remove the example and all its resources from the cluster: +Run the following command to remove the installation from the cluster: -1. Remove the stack by calling helm: +1. Remove the stack by calling Helm: ```bash helm delete -n ${KYMA_EXAMPLE_NS} ${HELM_RELEASE_NAME} diff --git a/prometheus/values.yaml b/prometheus/values.yaml index 415ff113..506dfc68 100644 --- a/prometheus/values.yaml +++ b/prometheus/values.yaml @@ -1,13 +1,13 @@ ####### This block is required to run in parallel with a Kyma monitoring stack prometheusOperator: - # exclude the kyma-system namespace from watching to not conflict with the kyma monitoring stack + # exclude the kyma-system Namespace from watching to avoid conflict with the Kyma monitoring stack denyNamespaces: - kyma-system kubeletService: - # re-use the kublet service definition of the kyma monitoring stack. + # re-use the kublet service definition of the Kyma monitoring stack. enabled: false prometheus-node-exporter: - # change the port of the node-export to be different from the one used by the kyma monitoring stack + # change the port of the node-export to be different from the one used by the Kyma monitoring stack service: port: 9101 targetPort: 9101 @@ -23,16 +23,16 @@ kubeProxy: kubeScheduler: enabled: false -####### This block is needed to also use ServiceMonitors which are not deployed as part of the chart +####### This block is needed to also use ServiceMonitors, which are not deployed as part of the chart prometheus: prometheusSpec: - # Disable requires helm release labels on ServiceMonitors + # Disable required Helm release labels on ServiceMonitors serviceMonitorSelectorNilUsesHelmValues: false - # Disable requires helm release labels on PodMonitors + # Disable required Helm release labels on PodMonitors podMonitorSelectorNilUsesHelmValues: false - # Disable requires helm release labels on Probes + # Disable required Helm release labels on Probes probeSelectorNilUsesHelmValues: false - # Disable requires helm release labels on Rules + # Disable required Helm release labels on Rules ruleSelectorNilUsesHelmValues: false ####### This block is required to enable scraping of endpoints with Istio strict mTLS, see also https://istio.io/latest/docs/ops/integrations/prometheus/#tls-settings From 97c84c8264f504fabc9e0edcb0233da53b502935 Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Fri, 23 Sep 2022 17:20:34 +0200 Subject: [PATCH 08/30] added istio support --- prometheus/README.md | 35 + .../configmap-istio-grafana-dashboards.yaml | 10 + ...map-istio-services-grafana-dashboards.yaml | 893 ++++++++++++++++++ prometheus/istio/podmonitor-istio-proxy.yaml | 13 + prometheus/istio/servicemonitor-istiod.yaml | 13 + prometheus/values.yaml | 137 ++- 6 files changed, 1087 insertions(+), 14 deletions(-) create mode 100644 prometheus/istio/configmap-istio-grafana-dashboards.yaml create mode 100644 prometheus/istio/configmap-istio-services-grafana-dashboards.yaml create mode 100644 prometheus/istio/podmonitor-istio-proxy.yaml create mode 100644 prometheus/istio/servicemonitor-istiod.yaml diff --git a/prometheus/README.md b/prometheus/README.md index 68d6331a..9ae91e4a 100644 --- a/prometheus/README.md +++ b/prometheus/README.md @@ -48,6 +48,28 @@ The provided `values.yaml` covers the following adjustments: - Support parallel operation to a Kyma monitoring stack - Support the scraping of workload secured with Istio strict mTLS +### Install Istio Support + +To configure Prometheus for scraping of the istio specific metrics from any istio-proxy running in the cluster, deploy a PodMonitor which scrapes any pod having a port with name `.*-envoy-prom` exposed. + +```bash +kubectl -n ${KYMA_EXAMPLE_NS} apply -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/istio/podmonitor-istio-proxy.yaml +``` + +Also deploy a ServiceMonitor definition for the central metrics of the `istiod` deployment. +```bash +kubectl -n ${KYMA_EXAMPLE_NS} apply -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/istio/servicemonitor-istiod.yaml +``` + +As Grafana is configured to load dashboards dynamically from ConfigMaps in the cluster, Istio specific dashboards can be applied as well. To get the latest versions you can follow the official [instructions](https://istio.io/latest/docs/ops/integrations/grafana/#option-1-quick-start) or you take the prepared ones by calling: + +```bash +kubectl -n ${KYMA_EXAMPLE_NS} apply -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/istio/configmap-istio-grafana-dashboards.yaml +kubectl -n ${KYMA_EXAMPLE_NS} apply -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/istio/configmap-istio-services-grafana-dashboards.yaml +``` + +Please have in mind that this setup will collect all istio metrics on a pod level. As this can lead to cardinality issues and metrics are only needed on service level, it is recommended to use an improved setup based on federation as described in the [istio documentation](https://istio.io/latest/docs/ops/best-practices/observability/#using-prometheus-for-production-scale-monitoring). + ### Verify the installation 1. You should see several Pods coming up in the Namespace, especially Prometheus and Alertmanager. Assure that all Pods have the "Running" state. @@ -64,6 +86,19 @@ The provided `values.yaml` covers the following adjustments: 1. Follow the tutorial [monitoring-custom-metrics](./../monitoring-custom-metrics/), but use the steps above to verify that the metrics are collected. +### Scrape workload via annotations + +Instead of defining a ServiceMonitor per workload for setting up custom metric scraping, you can use a simplified way based on annotations. The used [values.yaml](./values.yaml) defines an `additionalScrapeConfig` which will scrape all pods and services having these annotations: + +```yaml +prometheus.io/scrape: "true" # mandatory to enable automatic scraping +prometheus.io/scheme: https # optional, default is http. Use "https" to scrape using istio client certificates +prometheus.io/port: "1234" # optional, configure the port under which the metrics are exposed +prometheus.io/path: /myMetrics # optional, configure the path under which the metrics are exposed +``` + +You can try it out by removing the ServiceMonitor of the example used above and instead providing the annotations to the Service manifest. + ### Cleanup Run the following command to remove the installation from the cluster: diff --git a/prometheus/istio/configmap-istio-grafana-dashboards.yaml b/prometheus/istio/configmap-istio-grafana-dashboards.yaml new file mode 100644 index 00000000..dda91e3e --- /dev/null +++ b/prometheus/istio/configmap-istio-grafana-dashboards.yaml @@ -0,0 +1,10 @@ +apiVersion: v1 +data: + istio-performance-dashboard.json: | + {"annotations":{"list":[{"builtIn":1,"datasource":"-- Grafana --","enable":true,"hide":true,"iconColor":"rgba(0, 211, 255, 1)","name":"Annotations & Alerts","type":"dashboard"}]},"editable":false,"gnetId":null,"graphTooltip":0,"links":[],"panels":[{"collapsed":true,"gridPos":{"h":1,"w":24,"x":0,"y":0},"id":21,"panels":[{"content":"The charts on this dashboard are intended to show Istio main components cost in terms of resources utilization under steady load.\n\n- **vCPU / 1k rps:** shows vCPU utilization by the main Istio components normalized by 1000 requests/second. When idle or low traffic, this chart will be blank. The curve for istio-proxy refers to the services sidecars only.\n- **vCPU:** vCPU utilization by Istio components, not normalized.\n- **Memory:** memory footprint for the components. Telemetry and policy are normalized by 1k rps, and no data is shown when there is no traffic. For ingress and istio-proxy, the data is per instance.\n- **Bytes transferred / sec:** shows the number of bytes flowing through each Istio component.\n\n\n","gridPos":{"h":6,"w":24,"x":0,"y":1},"id":19,"links":[],"mode":"markdown","timeFrom":null,"timeShift":null,"title":"Performance Dashboard README","transparent":true,"type":"text"}],"title":"Performance Dashboard Notes","type":"row"},{"collapsed":false,"gridPos":{"h":1,"w":24,"x":0,"y":1},"id":6,"panels":[],"title":"vCPU Usage","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":8,"w":12,"x":0,"y":2},"id":4,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"(sum(irate(container_cpu_usage_seconds_total{pod=~\"istio-ingressgateway-.*\",container=\"istio-proxy\"}[1m])) / (round(sum(irate(istio_requests_total{source_workload=\"istio-ingressgateway\", reporter=\"source\"}[1m])), 0.001)/1000))","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"istio-ingressgateway","refId":"A"},{"expr":"(sum(irate(container_cpu_usage_seconds_total{namespace!=\"istio-system\",container=\"istio-proxy\"}[1m]))/ (round(sum(irate(istio_requests_total[1m])), 0.001)/1000))/ (sum(irate(istio_requests_total{source_workload=\"istio-ingressgateway\"}[1m])) >bool 10)","format":"time_series","intervalFactor":1,"legendFormat":"istio-proxy","refId":"B"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"vCPU / 1k rps","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":8,"w":12,"x":12,"y":2},"id":7,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(rate(container_cpu_usage_seconds_total{pod=~\"istio-ingressgateway-.*\",container=\"istio-proxy\"}[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"istio-ingressgateway","refId":"A"},{"expr":"sum(rate(container_cpu_usage_seconds_total{namespace!=\"istio-system\",container=\"istio-proxy\"}[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"istio-proxy","refId":"B"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"vCPU","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"gridPos":{"h":1,"w":24,"x":0,"y":10},"id":13,"panels":[],"title":"Memory and Data Rates","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":8,"w":12,"x":0,"y":11},"id":902,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(container_memory_working_set_bytes{pod=~\"istio-ingressgateway-.*\"}) / count(container_memory_working_set_bytes{pod=~\"istio-ingressgateway-.*\",container!=\"POD\"})","format":"time_series","intervalFactor":1,"legendFormat":"per istio-ingressgateway","refId":"A"},{"expr":"sum(container_memory_working_set_bytes{namespace!=\"istio-system\",container=\"istio-proxy\"}) / count(container_memory_working_set_bytes{namespace!=\"istio-system\",container=\"istio-proxy\"})","format":"time_series","intervalFactor":1,"legendFormat":"per istio proxy","refId":"B"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Memory Usage","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bytes","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":8,"w":12,"x":12,"y":11},"id":11,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(irate(istio_response_bytes_sum{source_workload=\"istio-ingressgateway\", reporter=\"source\"}[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"istio-ingressgateway","refId":"A"},{"expr":"sum(irate(istio_response_bytes_sum{source_workload_namespace!=\"istio-system\", reporter=\"source\"}[1m])) + sum(irate(istio_request_bytes_sum{source_workload_namespace!=\"istio-system\", reporter=\"source\"}[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"istio-proxy","refId":"B"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Bytes transferred / sec","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"Bps","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"gridPos":{"h":1,"w":24,"x":0,"y":19},"id":17,"panels":[],"title":"Istio Component Versions","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":8,"w":24,"x":0,"y":20},"id":15,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(istio_build) by (component, tag)","format":"time_series","intervalFactor":1,"legendFormat":"{{ component }}: {{ tag }}","refId":"A"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Istio Components by Version","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"gridPos":{"h":1,"w":24,"x":0,"y":31},"id":71,"panels":[],"title":"Proxy Resource Usage","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":7,"w":6,"x":0,"y":32},"id":72,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(container_memory_working_set_bytes{container=\"istio-proxy\"})","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"Total (k8s)","refId":"A","step":2}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Memory","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bytes","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":7,"w":6,"x":6,"y":32},"id":73,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(rate(container_cpu_usage_seconds_total{container=\"istio-proxy\"}[1m]))","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"Total (k8s)","refId":"A","step":2}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"vCPU","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":7,"w":6,"x":12,"y":32},"id":702,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(container_fs_usage_bytes{container=\"istio-proxy\"})","format":"time_series","intervalFactor":2,"legendFormat":"Total (k8s)","refId":"A","step":2}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Disk","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bytes","label":"","logBase":1,"max":null,"min":null,"show":true},{"decimals":null,"format":"none","label":"","logBase":1024,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"gridPos":{"h":1,"w":24,"x":0,"y":39},"id":69,"panels":[],"title":"Istiod Resource Usage","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":7,"w":6,"x":0,"y":40},"id":5,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"process_virtual_memory_bytes{app=\"istiod\"}","format":"time_series","instant":false,"intervalFactor":2,"legendFormat":"Virtual Memory","refId":"I","step":2},{"expr":"process_resident_memory_bytes{app=\"istiod\"}","format":"time_series","intervalFactor":2,"legendFormat":"Resident Memory","refId":"H","step":2},{"expr":"go_memstats_heap_sys_bytes{app=\"istiod\"}","format":"time_series","hide":true,"intervalFactor":2,"legendFormat":"heap sys","refId":"A"},{"expr":"go_memstats_heap_alloc_bytes{app=\"istiod\"}","format":"time_series","hide":true,"intervalFactor":2,"legendFormat":"heap alloc","refId":"D"},{"expr":"go_memstats_alloc_bytes{app=\"istiod\"}","format":"time_series","intervalFactor":2,"legendFormat":"Alloc","refId":"F","step":2},{"expr":"go_memstats_heap_inuse_bytes{app=\"istiod\"}","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"Heap in-use","refId":"E","step":2},{"expr":"go_memstats_stack_inuse_bytes{app=\"istiod\"}","format":"time_series","intervalFactor":2,"legendFormat":"Stack in-use","refId":"G","step":2},{"expr":"sum(container_memory_working_set_bytes{container=~\"discovery|istio-proxy\", pod=~\"istiod-.*\"})","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"Total (k8s)","refId":"C","step":2},{"expr":"container_memory_working_set_bytes{container=~\"discovery|istio-proxy\", pod=~\"istiod-.*\"}","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"{{ container }} (k8s)","refId":"B","step":2}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Memory","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bytes","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":7,"w":6,"x":6,"y":40},"id":602,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(rate(container_cpu_usage_seconds_total{container=~\"discovery|istio-proxy\", pod=~\"istiod-.*\"}[1m]))","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"Total (k8s)","refId":"A","step":2},{"expr":"sum(rate(container_cpu_usage_seconds_total{container=~\"discovery|istio-proxy\", pod=~\"istiod-.*\"}[1m])) by (container)","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"{{ container }} (k8s)","refId":"B","step":2},{"expr":"irate(process_cpu_seconds_total{app=\"istiod\"}[1m])","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"pilot (self-reported)","refId":"C","step":2}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"vCPU","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":7,"w":6,"x":12,"y":40},"id":74,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"process_open_fds{app=\"istiod\"}","format":"time_series","hide":true,"instant":false,"interval":"","intervalFactor":2,"legendFormat":"Open FDs (pilot)","refId":"A"},{"expr":"container_fs_usage_bytes{ container=~\"discovery|istio-proxy\", pod=~\"istiod-.*\"}","format":"time_series","intervalFactor":2,"legendFormat":"{{ container }}","refId":"B","step":2}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Disk","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bytes","label":"","logBase":1,"max":null,"min":null,"show":true},{"decimals":null,"format":"none","label":"","logBase":1024,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":7,"w":6,"x":18,"y":40},"id":402,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":false,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"go_goroutines{app=\"istiod\"}","format":"time_series","intervalFactor":2,"legendFormat":"Number of Goroutines","refId":"A","step":2}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Goroutines","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":"","logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}}],"refresh":"10s","schemaVersion":18,"style":"dark","tags":[],"templating":{"list":[]},"time":{"from":"now-5m","to":"now"},"timepicker":{"refresh_intervals":["5s","10s","30s","1m","5m","15m","30m","1h","2h","1d"],"time_options":["5m","15m","1h","6h","12h","24h","2d","7d","30d"]},"timezone":"","title":"Istio Performance Dashboard","uid":"vu8e0VWZk","version":22} + pilot-dashboard.json: | + {"annotations":{"list":[{"builtIn":1,"datasource":"-- Grafana --","enable":true,"hide":true,"iconColor":"rgba(0, 211, 255, 1)","name":"Annotations & Alerts","type":"dashboard"}]},"editable":false,"gnetId":null,"graphTooltip":1,"links":[],"panels":[{"collapsed":false,"gridPos":{"h":1,"w":24,"x":0,"y":0},"id":60,"panels":[],"title":"Deployed Versions","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":5,"w":24,"x":0,"y":1},"id":56,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(istio_build{component=\"pilot\"}) by (tag)","format":"time_series","intervalFactor":1,"legendFormat":"{{ tag }}","refId":"A"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Pilot Versions","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"gridPos":{"h":1,"w":24,"x":0,"y":6},"id":62,"panels":[],"title":"Resource Usage","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":7,"w":6,"x":0,"y":7},"id":5,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"process_virtual_memory_bytes{app=\"istiod\"}","format":"time_series","instant":false,"intervalFactor":2,"legendFormat":"Virtual Memory","refId":"I","step":2},{"expr":"process_resident_memory_bytes{app=\"istiod\"}","format":"time_series","intervalFactor":2,"legendFormat":"Resident Memory","refId":"H","step":2},{"expr":"go_memstats_heap_sys_bytes{app=\"istiod\"}","format":"time_series","hide":true,"intervalFactor":2,"legendFormat":"heap sys","refId":"A"},{"expr":"go_memstats_heap_alloc_bytes{app=\"istiod\"}","format":"time_series","hide":true,"intervalFactor":2,"legendFormat":"heap alloc","refId":"D"},{"expr":"go_memstats_alloc_bytes{app=\"istiod\"}","format":"time_series","intervalFactor":2,"legendFormat":"Alloc","refId":"F","step":2},{"expr":"go_memstats_heap_inuse_bytes{app=\"istiod\"}","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"Heap in-use","refId":"E","step":2},{"expr":"go_memstats_stack_inuse_bytes{app=\"istiod\"}","format":"time_series","intervalFactor":2,"legendFormat":"Stack in-use","refId":"G","step":2},{"expr":"container_memory_working_set_bytes{container=~\"discovery\", pod=~\"istiod-.*|istio-pilot-.*\"}","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"Discovery (container)","refId":"B","step":2},{"expr":"container_memory_working_set_bytes{container=~\"istio-proxy\", pod=~\"istiod-.*|istio-pilot-.*\"}","format":"time_series","intervalFactor":1,"legendFormat":"Sidecar (container)","refId":"C"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Memory","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bytes","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":7,"w":6,"x":6,"y":7},"id":6,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(irate(container_cpu_usage_seconds_total{container=\"discovery\", pod=~\"istiod-.*|istio-pilot-.*\"}[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"Discovery (container)","refId":"A"},{"expr":"irate(process_cpu_seconds_total{app=\"istiod\"}[1m])","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"Discovery (process)","refId":"C","step":2},{"expr":"sum(irate(container_cpu_usage_seconds_total{container=\"istio-proxy\", pod=~\"istiod-.*|istio-pilot-.*\"}[1m]))","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"Sidecar (container)","refId":"B","step":2}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"CPU","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":7,"w":6,"x":12,"y":7},"id":7,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"container_fs_usage_bytes{container=\"discovery\", pod=~\"istiod-.*|istio-pilot-.*\"}","format":"time_series","intervalFactor":2,"legendFormat":"Discovery","refId":"B","step":2},{"expr":"container_fs_usage_bytes{container=\"istio-proxy\", pod=~\"istiod-.*|istio-pilot-.*\"}","format":"time_series","intervalFactor":1,"legendFormat":"Sidecar","refId":"A"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Disk","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bytes","label":"","logBase":1,"max":null,"min":null,"show":true},{"decimals":null,"format":"none","label":"","logBase":1024,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":7,"w":6,"x":18,"y":7},"id":4,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":false,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"go_goroutines{app=\"istiod\"}","format":"time_series","intervalFactor":2,"legendFormat":"Number of Goroutines","refId":"A","step":2}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Goroutines","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":"","logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"gridPos":{"h":1,"w":24,"x":0,"y":14},"id":58,"panels":[],"title":"Pilot Push Information","type":"row"},{"aliasColors":{},"bars":true,"dashLength":10,"dashes":false,"datasource":"Prometheus","description":"Shows the rate of pilot pushes","fill":1,"gridPos":{"h":8,"w":8,"x":0,"y":15},"id":622,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":false,"linewidth":1,"links":[],"nullPointMode":"null as zero","paceLength":10,"percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":true,"steppedLine":false,"targets":[{"expr":"sum(irate(pilot_xds_pushes{type=\"cds\"}[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"Cluster","refId":"C"},{"expr":"sum(irate(pilot_xds_pushes{type=\"eds\"}[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"Endpoints","refId":"D"},{"expr":"sum(irate(pilot_xds_pushes{type=\"lds\"}[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"Listeners","refId":"A"},{"expr":"sum(irate(pilot_xds_pushes{type=\"rds\"}[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"Routes","refId":"E"},{"expr":"sum(irate(pilot_xds_pushes{type=\"sds\"}[1m]))","interval":"","legendFormat":"Secrets","refId":"B"},{"expr":"sum(irate(pilot_xds_pushes{type=\"nds\"}[1m]))","interval":"","legendFormat":"Nametables","refId":"F"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Pilot Pushes","tooltip":{"shared":false,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":["total"]},"yaxes":[{"format":"ops","label":null,"logBase":1,"max":null,"min":"0","show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","description":"Captures a variety of pilot errors","fill":1,"gridPos":{"h":8,"w":8,"x":8,"y":15},"id":67,"legend":{"avg":false,"current":false,"hideEmpty":true,"hideZero":true,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(pilot_xds_cds_reject{app=\"istiod\"}) or (absent(pilot_xds_cds_reject{app=\"istiod\"}) - 1)","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"Rejected CDS Configs","refId":"C"},{"expr":"sum(pilot_xds_eds_reject{app=\"istiod\"}) or (absent(pilot_xds_eds_reject{app=\"istiod\"}) - 1)","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"Rejected EDS Configs","refId":"D"},{"expr":"sum(pilot_xds_rds_reject{app=\"istiod\"}) or (absent(pilot_xds_rds_reject{app=\"istiod\"}) - 1)","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"Rejected RDS Configs","refId":"A"},{"expr":"sum(pilot_xds_lds_reject{app=\"istiod\"}) or (absent(pilot_xds_lds_reject{app=\"istiod\"}) - 1)","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"Rejected LDS Configs","refId":"B"},{"expr":"sum(rate(pilot_xds_write_timeout{app=\"istiod\"}[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"Write Timeouts","refId":"F"},{"expr":"sum(rate(pilot_total_xds_internal_errors{app=\"istiod\"}[1m]))","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"Internal Errors","refId":"H"},{"expr":"sum(rate(pilot_total_xds_rejects{app=\"istiod\"}[1m]))","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"Config Rejection Rate","refId":"E"},{"expr":"sum(rate(pilot_xds_push_context_errors{app=\"istiod\"}[1m]))","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"Push Context Errors","refId":"K"},{"expr":"sum(rate(pilot_xds_write_timeout{app=\"istiod\"}[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"Push Timeouts","refId":"G"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Pilot Errors","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","description":"Shows the total time it takes to push a config update to a proxy","fill":1,"gridPos":{"h":8,"w":8,"x":16,"y":15},"id":624,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"histogram_quantile(0.5, sum(rate(pilot_proxy_convergence_time_bucket[1m])) by (le))","format":"time_series","intervalFactor":1,"legendFormat":"p50 ","refId":"A"},{"expr":"histogram_quantile(0.9, sum(rate(pilot_proxy_convergence_time_bucket[1m])) by (le))","format":"time_series","intervalFactor":1,"legendFormat":"p90","refId":"B"},{"expr":"histogram_quantile(0.99, sum(rate(pilot_proxy_convergence_time_bucket[1m])) by (le))","format":"time_series","intervalFactor":1,"legendFormat":"p99","refId":"C"},{"expr":"histogram_quantile(0.999, sum(rate(pilot_proxy_convergence_time_bucket[1m])) by (le))","format":"time_series","intervalFactor":1,"legendFormat":"p99.9","refId":"D"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Proxy Push Time","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"s","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":8,"w":12,"x":0,"y":23},"id":45,"legend":{"avg":false,"current":false,"hideEmpty":true,"hideZero":true,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null as zero","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"pilot_conflict_inbound_listener{app=\"istiod\"}","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"Inbound Listeners","refId":"B"},{"expr":"pilot_conflict_outbound_listener_http_over_current_tcp{app=\"istiod\"}","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"Outbound Listeners (http over current tcp)","refId":"A"},{"expr":"pilot_conflict_outbound_listener_tcp_over_current_tcp{app=\"istiod\"}","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"Outbound Listeners (tcp over current tcp)","refId":"C"},{"expr":"pilot_conflict_outbound_listener_tcp_over_current_http{app=\"istiod\"}","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"Outbound Listeners (tcp over current http)","refId":"D"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Conflicts","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":8,"w":12,"x":12,"y":23},"id":47,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"avg(pilot_virt_services{app=\"istiod\"})","format":"time_series","intervalFactor":1,"legendFormat":"Virtual Services","refId":"A"},{"expr":"avg(pilot_services{app=\"istiod\"})","format":"time_series","intervalFactor":1,"legendFormat":"Services","refId":"B"},{"expr":"sum(pilot_xds{app=\"istiod\"}) by (pod)","format":"time_series","intervalFactor":1,"legendFormat":"Connected Endpoints {{pod}}","refId":"E"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"ADS Monitoring","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"gridPos":{"h":1,"w":24,"x":0,"y":31},"id":64,"panels":[],"title":"Envoy Information","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","description":"Shows details about Envoy proxies in the mesh","fill":1,"gridPos":{"h":8,"w":8,"x":0,"y":32},"id":40,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(irate(envoy_cluster_upstream_cx_total{cluster_name=\"xds-grpc\"}[1m]))","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"XDS Connections","refId":"C"},{"expr":"sum(irate(envoy_cluster_upstream_cx_connect_fail{cluster_name=\"xds-grpc\"}[1m]))","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"XDS Connection Failures","refId":"A"},{"expr":"sum(increase(envoy_server_hot_restart_epoch[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"Envoy Restarts","refId":"B"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Envoy Details","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"ops","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"ops","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":8,"w":8,"x":8,"y":32},"id":41,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(envoy_cluster_upstream_cx_active{cluster_name=\"xds-grpc\"})","format":"time_series","intervalFactor":2,"legendFormat":"XDS Active Connections","refId":"C","step":2}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"XDS Active Connections","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","description":"Shows the size of XDS requests and responses","fill":1,"gridPos":{"h":8,"w":8,"x":16,"y":32},"id":42,"legend":{"avg":false,"current":false,"hideEmpty":false,"hideZero":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"max(rate(envoy_cluster_upstream_cx_rx_bytes_total{cluster_name=\"xds-grpc\"}[1m]))","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"XDS Response Bytes Max","refId":"D"},{"expr":"quantile(0.5, rate(envoy_cluster_upstream_cx_rx_bytes_total{cluster_name=\"xds-grpc\"}[1m]))","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"XDS Response Bytes Average","refId":"B"},{"expr":"max(rate(envoy_cluster_upstream_cx_tx_bytes_total{cluster_name=\"xds-grpc\"}[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"XDS Request Bytes Max","refId":"A"},{"expr":"quantile(.5, rate(envoy_cluster_upstream_cx_tx_bytes_total{cluster_name=\"xds-grpc\"}[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"XDS Request Bytes Average","refId":"C"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"XDS Requests Size","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"Bps","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"ops","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"datasource":null,"gridPos":{"h":1,"w":24,"x":0,"y":40},"id":626,"panels":[],"title":"Webhooks","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":null,"fill":1,"fillGradient":0,"gridPos":{"h":8,"w":12,"x":0,"y":41},"hiddenSeries":false,"id":629,"legend":{"avg":false,"current":false,"hideEmpty":false,"hideZero":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"nullPointMode":"null","options":{"dataLinks":[]},"percentage":false,"pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(rate(galley_validation_passed[1m]))","interval":"","legendFormat":"Validations (Success)","refId":"A"},{"expr":"sum(rate(galley_validation_failed[1m]))","interval":"","legendFormat":"Validation (Failure)","refId":"B"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Configuration Validation","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":null,"description":"","fill":1,"fillGradient":0,"gridPos":{"h":8,"w":12,"x":12,"y":41},"hiddenSeries":false,"id":630,"legend":{"avg":false,"current":false,"hideZero":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"nullPointMode":"null","options":{"dataLinks":[]},"percentage":false,"pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(rate(sidecar_injection_success_total[1m]))","interval":"","legendFormat":"Injections (Success)","refId":"A"},{"expr":"sum(rate(sidecar_injection_failure_total[1m]))","interval":"","legendFormat":"Injections (Failure)","refId":"B"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Sidecar Injection","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}}],"refresh":"5s","schemaVersion":18,"style":"dark","tags":[],"templating":{"list":[{"current":{"selected":true,"text":"default","value":"default"},"hide":0,"includeAll":false,"label":null,"multi":false,"name":"datasource","options":[],"query":"prometheus","queryValue":"","refresh":1,"regex":"","skipUrlSync":false,"type":"datasource"}]},"time":{"from":"now-5m","to":"now"},"timepicker":{"refresh_intervals":["5s","10s","30s","1m","5m","15m","30m","1h","2h","1d"],"time_options":["5m","15m","1h","6h","12h","24h","2d","7d","30d"]},"timezone":"browser","title":"Istio Control Plane Dashboard","uid":"3--MLVZZk","version":11} +kind: ConfigMap +metadata: + creationTimestamp: null + name: istio-grafana-dashboards diff --git a/prometheus/istio/configmap-istio-services-grafana-dashboards.yaml b/prometheus/istio/configmap-istio-services-grafana-dashboards.yaml new file mode 100644 index 00000000..369c36ea --- /dev/null +++ b/prometheus/istio/configmap-istio-services-grafana-dashboards.yaml @@ -0,0 +1,893 @@ +apiVersion: v1 +data: + istio-extension-dashboard.json: | + {"annotations":{"list":[{"builtIn":1,"datasource":"-- Grafana --","enable":true,"hide":true,"iconColor":"rgba(0, 211, 255, 1)","name":"Annotations & Alerts","type":"dashboard"}]},"editable":false,"gnetId":null,"graphTooltip":0,"links":[],"panels":[{"collapsed":false,"datasource":"Prometheus","gridPos":{"h":1,"w":24,"x":0,"y":0},"id":3,"panels":[],"title":"Wasm VMs","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","description":"","fieldConfig":{"defaults":{"custom":{"align":null},"links":[],"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]}},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":8,"w":12,"x":0,"y":1},"hiddenSeries":false,"id":2,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"nullPointMode":"null","options":{"alertThreshold":true},"percentage":false,"pluginVersion":"7.2.1","pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"avg(envoy_wasm_envoy_wasm_runtime_null_active)","interval":"","legendFormat":"native","refId":"A"},{"expr":"avg(envoy_wasm_envoy_wasm_runtime_v8_active)","interval":"","legendFormat":"v8","refId":"B"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Active","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"$$hashKey":"object:123","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"$$hashKey":"object:124","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fieldConfig":{"defaults":{"custom":{},"links":[]},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":8,"w":12,"x":12,"y":1},"hiddenSeries":false,"id":6,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"nullPointMode":"null","options":{"alertThreshold":true},"percentage":false,"pluginVersion":"7.2.1","pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"avg(envoy_wasm_envoy_wasm_runtime_null_created)","interval":"","legendFormat":"native","refId":"A"},{"expr":"avg(envoy_wasm_envoy_wasm_runtime_v8_created)","interval":"","legendFormat":"v8","refId":"B"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Created","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"$$hashKey":"object:68","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"$$hashKey":"object:69","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"datasource":"Prometheus","gridPos":{"h":1,"w":24,"x":0,"y":9},"id":7,"panels":[],"title":"Wasm Module Remote Load","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fieldConfig":{"defaults":{"custom":{},"links":[]},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":8,"w":8,"x":0,"y":10},"hiddenSeries":false,"id":11,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"nullPointMode":"null","options":{"alertThreshold":true},"percentage":false,"pluginVersion":"7.2.1","pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"avg(envoy_wasm_remote_load_cache_entries)","interval":"","legendFormat":"entries","refId":"A"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Cache Entry","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"$$hashKey":"object:178","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"$$hashKey":"object:179","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fieldConfig":{"defaults":{"custom":{},"links":[]},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":8,"w":8,"x":8,"y":10},"hiddenSeries":false,"id":8,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"nullPointMode":"null","options":{"alertThreshold":true},"percentage":false,"pluginVersion":"7.2.1","pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"avg(envoy_wasm_remote_load_cache_hits)","interval":"","legendFormat":"hits","refId":"A"},{"expr":"avg(envoy_wasm_remote_load_cache_misses)","interval":"","legendFormat":"misses","refId":"B"},{"expr":"avg(envoy_wasm_remote_load_cache_negative_hits)","interval":"","legendFormat":"negative hits","refId":"C"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Cache Visit","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"$$hashKey":"object:233","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"$$hashKey":"object:234","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fieldConfig":{"defaults":{"custom":{},"links":[]},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":8,"w":8,"x":16,"y":10},"hiddenSeries":false,"id":10,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"nullPointMode":"null","options":{"alertThreshold":true},"percentage":false,"pluginVersion":"7.2.1","pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"avg(envoy_wasm_remote_load_fetch_failures)","interval":"","legendFormat":"failures","refId":"A"},{"expr":"avg(envoy_wasm_remote_load_fetch_successes)","interval":"","legendFormat":"successes","refId":"B"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Remote Fetch","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"$$hashKey":"object:288","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"$$hashKey":"object:289","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"datasource":"Prometheus","gridPos":{"h":1,"w":24,"x":0,"y":18},"id":71,"panels":[],"title":"Proxy Resource Usage","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":8,"w":12,"x":0,"y":19},"hiddenSeries":false,"id":72,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","options":{"alertThreshold":true},"percentage":false,"pluginVersion":"7.2.1","pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(container_memory_working_set_bytes{container=\"istio-proxy\"})","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"Total (k8s)","refId":"A","step":2}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Memory","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"$$hashKey":"object:396","format":"bytes","label":null,"logBase":1,"max":null,"min":null,"show":true},{"$$hashKey":"object:397","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":8,"w":12,"x":12,"y":19},"hiddenSeries":false,"id":73,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","options":{"alertThreshold":true},"percentage":false,"pluginVersion":"7.2.1","pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(rate(container_cpu_usage_seconds_total{container=\"istio-proxy\"}[1m]))","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"Total (k8s)","refId":"A","step":2}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"vCPU","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"$$hashKey":"object:447","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"$$hashKey":"object:448","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}}],"refresh":false,"schemaVersion":26,"style":"dark","tags":[],"templating":{"list":[]},"time":{"from":"now-5m","to":"now"},"timepicker":{"refresh_intervals":["10s","30s","1m","5m","15m","30m","1h","2h","1d"]},"timezone":"","title":"Istio Wasm Extension Dashboard","uid":"7PAV7ctGz","version":17} + istio-mesh-dashboard.json: | + {"annotations":{"list":[{"builtIn":1,"datasource":"-- Grafana --","enable":true,"hide":true,"iconColor":"rgba(0, 211, 255, 1)","name":"Annotations & Alerts","type":"dashboard"}]},"editable":false,"gnetId":null,"graphTooltip":0,"id":null,"links":[],"panels":[{"content":"
\n
\n Istio\n
\n
\n Istio is an open platform that provides a uniform way to secure,\n connect, and \n monitor microservices.\n
\n Need help? Join the Istio community.\n
\n
","gridPos":{"h":3,"w":24,"x":0,"y":0},"height":"50px","id":13,"links":[],"mode":"html","style":{"font-size":"18pt"},"title":"","transparent":true,"type":"text"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource":"Prometheus","format":"ops","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":6,"x":0,"y":3},"id":20,"interval":null,"links":[],"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"horizontal","reduceOptions":{"calcs":["lastNotNull"],"fields":"","values":false},"textMode":"auto"},"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":true,"lineColor":"rgb(31, 120, 193)","show":true},"tableColumn":"","targets":[{"expr":"round(sum(irate(istio_requests_total{reporter=\"source\"}[1m])), 0.001)","intervalFactor":1,"refId":"A","step":4}],"thresholds":"","title":"Global Request Volume","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"avg"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource":"Prometheus","format":"percentunit","gauge":{"maxValue":100,"minValue":80,"show":false,"thresholdLabels":false,"thresholdMarkers":false},"gridPos":{"h":3,"w":6,"x":6,"y":3},"id":21,"interval":null,"links":[],"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"horizontal","reduceOptions":{"calcs":["lastNotNull"],"fields":"","values":false},"textMode":"auto"},"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":true,"lineColor":"rgb(31, 120, 193)","show":true},"tableColumn":"","targets":[{"expr":"sum(rate(istio_requests_total{reporter=\"source\", response_code!~\"5.*\"}[1m])) / sum(rate(istio_requests_total{reporter=\"source\"}[1m]))","format":"time_series","intervalFactor":1,"refId":"A","step":4}],"thresholds":"95, 99, 99.5","title":"Global Success Rate (non-5xx responses)","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"avg"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource":"Prometheus","format":"ops","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":6,"x":12,"y":3},"id":22,"interval":null,"links":[],"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"horizontal","reduceOptions":{"calcs":["lastNotNull"],"fields":"","values":false},"textMode":"auto"},"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":true,"lineColor":"rgb(31, 120, 193)","show":true},"tableColumn":"","targets":[{"expr":"sum(irate(istio_requests_total{reporter=\"source\", response_code=~\"4.*\"}[1m]))","format":"time_series","intervalFactor":1,"refId":"A","step":4}],"thresholds":"","title":"4xxs","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"avg"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"datasource":"Prometheus","format":"ops","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":6,"x":18,"y":3},"id":23,"interval":null,"links":[],"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"horizontal","reduceOptions":{"calcs":["lastNotNull"],"fields":"","values":false},"textMode":"auto"},"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":true,"lineColor":"rgb(31, 120, 193)","show":true},"tableColumn":"","targets":[{"expr":"sum(irate(istio_requests_total{reporter=\"source\", response_code=~\"5.*\"}[1m]))","format":"time_series","intervalFactor":1,"refId":"A","step":4}],"thresholds":"","title":"5xxs","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"avg"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["#299c46","rgba(237, 129, 40, 0.89)","#d44a3a"],"datasource":"Prometheus","format":"none","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":6,"x":0,"y":6},"id":113,"interval":null,"links":[],"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"horizontal","reduceOptions":{"calcs":["lastNotNull"],"fields":"","values":false},"textMode":"auto"},"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":true},"tableColumn":"","targets":[{"expr":"max(pilot_k8s_cfg_events{type=\"VirtualService\", event=\"add\"}) - (max(pilot_k8s_cfg_events{type=\"VirtualService\", event=\"delete\"}) or max(up * 0))","format":"time_series","intervalFactor":1,"refId":"A"}],"thresholds":"","timeFrom":null,"timeShift":null,"title":"Virtual Services","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"current"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["#299c46","rgba(237, 129, 40, 0.89)","#d44a3a"],"datasource":"Prometheus","format":"none","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":6,"x":6,"y":6},"id":114,"interval":null,"links":[],"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"horizontal","reduceOptions":{"calcs":["lastNotNull"],"fields":"","values":false},"textMode":"auto"},"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":true},"tableColumn":"","targets":[{"expr":"max(pilot_k8s_cfg_events{type=\"DestinationRule\", event=\"add\"}) - (max(pilot_k8s_cfg_events{type=\"DestinationRule\", event=\"delete\"}) or max(up * 0))","format":"time_series","intervalFactor":1,"refId":"A"}],"thresholds":"","timeFrom":null,"timeShift":null,"title":"Destination Rules","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"current"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["#299c46","rgba(237, 129, 40, 0.89)","#d44a3a"],"datasource":"Prometheus","format":"none","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":6,"x":12,"y":6},"id":115,"interval":null,"links":[],"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"horizontal","reduceOptions":{"calcs":["lastNotNull"],"fields":"","values":false},"textMode":"auto"},"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":true},"tableColumn":"","targets":[{"expr":"max(pilot_k8s_cfg_events{type=\"Gateway\", event=\"add\"}) - (max(pilot_k8s_cfg_events{type=\"Gateway\", event=\"delete\"}) or max(up * 0))","format":"time_series","intervalFactor":1,"refId":"A"}],"thresholds":"","timeFrom":null,"timeShift":null,"title":"Gateways","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"current"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["#299c46","rgba(237, 129, 40, 0.89)","#d44a3a"],"datasource":"Prometheus","format":"none","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":6,"x":18,"y":6},"id":116,"interval":null,"links":[],"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"horizontal","reduceOptions":{"calcs":["lastNotNull"],"fields":"","values":false},"textMode":"auto"},"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":true},"tableColumn":"","targets":[{"expr":"max(pilot_k8s_cfg_events{type=\"WorkloadEntry\", event=\"add\"}) - (max(pilot_k8s_cfg_events{type=\"WorkloadEntry\", event=\"delete\"}) or max(up * 0))","format":"time_series","intervalFactor":1,"refId":"A"}],"thresholds":"","timeFrom":null,"timeShift":null,"title":"Workload Entries","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"current"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["#299c46","rgba(237, 129, 40, 0.89)","#d44a3a"],"datasource":"Prometheus","format":"none","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":6,"x":0,"y":6},"id":117,"interval":null,"links":[],"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"horizontal","reduceOptions":{"calcs":["lastNotNull"],"fields":"","values":false},"textMode":"auto"},"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":true},"tableColumn":"","targets":[{"expr":"max(pilot_k8s_cfg_events{type=\"ServiceEntry\", event=\"add\"}) - (max(pilot_k8s_cfg_events{type=\"ServiceEntry\", event=\"delete\"}) or max(up * 0))","format":"time_series","intervalFactor":1,"refId":"A"}],"thresholds":"","timeFrom":null,"timeShift":null,"title":"Service Entries","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"current"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["#299c46","rgba(237, 129, 40, 0.89)","#d44a3a"],"datasource":"Prometheus","format":"none","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":6,"x":6,"y":6},"id":90,"interval":null,"links":[],"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"horizontal","reduceOptions":{"calcs":["lastNotNull"],"fields":"","values":false},"textMode":"auto"},"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":true},"tableColumn":"","targets":[{"expr":"max(pilot_k8s_cfg_events{type=\"PeerAuthentication\", event=\"add\"}) - (max(pilot_k8s_cfg_events{type=\"PeerAuthentication\", event=\"delete\"}) or max(up * 0))","format":"time_series","intervalFactor":1,"refId":"A"}],"thresholds":"","timeFrom":null,"timeShift":null,"title":"PeerAuthentication Policies","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"current"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["#299c46","rgba(237, 129, 40, 0.89)","#d44a3a"],"datasource":"Prometheus","format":"none","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":6,"x":12,"y":6},"id":91,"interval":null,"links":[],"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"horizontal","reduceOptions":{"calcs":["lastNotNull"],"fields":"","values":false},"textMode":"auto"},"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":true},"tableColumn":"","targets":[{"expr":"max(pilot_k8s_cfg_events{type=\"RequestAuthentication\", event=\"add\"}) - (max(pilot_k8s_cfg_events{type=\"RequestAuthentication\", event=\"delete\"}) or max(up * 0))","format":"time_series","intervalFactor":1,"refId":"A"}],"thresholds":"","timeFrom":null,"timeShift":null,"title":"RequestAuthentication Policies","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"current"},{"cacheTimeout":null,"colorBackground":false,"colorValue":false,"colors":["#299c46","rgba(237, 129, 40, 0.89)","#d44a3a"],"datasource":"Prometheus","format":"none","gauge":{"maxValue":100,"minValue":0,"show":false,"thresholdLabels":false,"thresholdMarkers":true},"gridPos":{"h":3,"w":6,"x":18,"y":6},"id":92,"interval":null,"links":[],"options":{"colorMode":"value","graphMode":"area","justifyMode":"auto","orientation":"horizontal","reduceOptions":{"calcs":["lastNotNull"],"fields":"","values":false},"textMode":"auto"},"mappingType":1,"mappingTypes":[{"name":"value to text","value":1},{"name":"range to text","value":2}],"maxDataPoints":100,"nullPointMode":"connected","nullText":null,"postfix":"","postfixFontSize":"50%","prefix":"","prefixFontSize":"50%","rangeMaps":[{"from":"null","text":"N/A","to":"null"}],"sparkline":{"fillColor":"rgba(31, 118, 189, 0.18)","full":false,"lineColor":"rgb(31, 120, 193)","show":true},"tableColumn":"","targets":[{"expr":"max(pilot_k8s_cfg_events{type=\"AuthorizationPolicy\", event=\"add\"}) - (max(pilot_k8s_cfg_events{type=\"AuthorizationPolicy\", event=\"delete\"}) or max(up * 0))","format":"time_series","intervalFactor":1,"refId":"A"}],"thresholds":"","timeFrom":null,"timeShift":null,"title":"Authorization Policies","type":"singlestat","valueFontSize":"80%","valueMaps":[{"op":"=","text":"N/A","value":"null"}],"valueName":"current"},{"columns":[],"datasource":"Prometheus","fontSize":"100%","gridPos":{"h":21,"w":24,"x":0,"y":9},"hideTimeOverride":false,"id":73,"links":[],"pageSize":null,"repeat":null,"repeatDirection":"v","scroll":true,"showHeader":true,"sort":{"col":5,"desc":true},"styles":[{"alias":"Workload","colorMode":null,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"dateFormat":"YYYY-MM-DD HH:mm:ss","decimals":2,"link":false,"linkTargetBlank":false,"linkTooltip":"Workload dashboard","linkUrl":"/dashboard/db/istio-workload-dashboard?var-namespace=${__cell_3:raw}&var-workload=${__cell_2:raw}","pattern":"destination_workload","preserveFormat":false,"sanitize":false,"thresholds":[],"type":"hidden","unit":"short"},{"alias":"","colorMode":null,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"dateFormat":"YYYY-MM-DD HH:mm:ss","decimals":2,"pattern":"Time","thresholds":[],"type":"hidden","unit":"short"},{"alias":"Requests","colorMode":null,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"dateFormat":"YYYY-MM-DD HH:mm:ss","decimals":2,"pattern":"Value #A","thresholds":[],"type":"number","unit":"ops"},{"alias":"P50 Latency","colorMode":null,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"dateFormat":"YYYY-MM-DD HH:mm:ss","decimals":2,"pattern":"Value #B","thresholds":[],"type":"number","unit":"s"},{"alias":"P90 Latency","colorMode":null,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"dateFormat":"YYYY-MM-DD HH:mm:ss","decimals":2,"pattern":"Value #C","thresholds":[],"type":"number","unit":"s"},{"alias":"P99 Latency","colorMode":null,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"dateFormat":"YYYY-MM-DD HH:mm:ss","decimals":2,"pattern":"Value #D","thresholds":[],"type":"number","unit":"s"},{"alias":"Success Rate","colorMode":"cell","colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"dateFormat":"YYYY-MM-DD HH:mm:ss","decimals":2,"pattern":"Value #E","thresholds":[".95"," 1.00"],"type":"number","unit":"percentunit"},{"alias":"Workload","colorMode":null,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"dateFormat":"YYYY-MM-DD HH:mm:ss","decimals":2,"link":true,"linkTooltip":"$__cell dashboard","linkUrl":"/dashboard/db/istio-workload-dashboard?var-workload=${__cell_2:raw}&var-namespace=${__cell_3:raw}","pattern":"destination_workload_var","thresholds":[],"type":"number","unit":"short"},{"alias":"Service","colorMode":null,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"dateFormat":"YYYY-MM-DD HH:mm:ss","decimals":2,"link":true,"linkTooltip":"$__cell dashboard","linkUrl":"/dashboard/db/istio-service-dashboard?var-service=${__cell_1:raw}","pattern":"destination_service","thresholds":[],"type":"string","unit":"short"},{"alias":"","colorMode":null,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"dateFormat":"YYYY-MM-DD HH:mm:ss","decimals":2,"pattern":"destination_workload_namespace","thresholds":[],"type":"hidden","unit":"short"}],"targets":[{"expr":"label_join(sum(rate(istio_requests_total{reporter=\"source\", response_code=\"200\"}[1m])) by (destination_workload, destination_workload_namespace, destination_service), \"destination_workload_var\", \".\", \"destination_workload\", \"destination_workload_namespace\")","format":"table","hide":false,"instant":true,"intervalFactor":1,"legendFormat":"{{ destination_workload}}.{{ destination_workload_namespace }}","refId":"A"},{"expr":"label_join((histogram_quantile(0.50, sum(rate(istio_request_duration_milliseconds_bucket{reporter=\"source\"}[1m])) by (le, destination_workload, destination_workload_namespace)) / 1000) or histogram_quantile(0.50, sum(rate(istio_request_duration_seconds_bucket{reporter=\"source\"}[1m])) by (le, destination_workload, destination_workload_namespace)), \"destination_workload_var\", \".\", \"destination_workload\", \"destination_workload_namespace\")","format":"table","hide":false,"instant":true,"intervalFactor":1,"legendFormat":"{{ destination_workload}}.{{ destination_workload_namespace }}","refId":"B"},{"expr":"label_join((histogram_quantile(0.90, sum(rate(istio_request_duration_milliseconds_bucket{reporter=\"source\"}[1m])) by (le, destination_workload, destination_workload_namespace)) / 1000) or histogram_quantile(0.90, sum(rate(istio_request_duration_seconds_bucket{reporter=\"source\"}[1m])) by (le, destination_workload, destination_workload_namespace)), \"destination_workload_var\", \".\", \"destination_workload\", \"destination_workload_namespace\")","format":"table","hide":false,"instant":true,"intervalFactor":1,"legendFormat":"{{ destination_workload }}.{{ destination_workload_namespace }}","refId":"C"},{"expr":"label_join((histogram_quantile(0.99, sum(rate(istio_request_duration_milliseconds_bucket{reporter=\"source\"}[1m])) by (le, destination_workload, destination_workload_namespace)) / 1000) or histogram_quantile(0.99, sum(rate(istio_request_duration_seconds_bucket{reporter=\"source\"}[1m])) by (le, destination_workload, destination_workload_namespace)), \"destination_workload_var\", \".\", \"destination_workload\", \"destination_workload_namespace\")","format":"table","hide":false,"instant":true,"intervalFactor":1,"legendFormat":"{{ destination_workload }}.{{ destination_workload_namespace }}","refId":"D"},{"expr":"label_join((sum(rate(istio_requests_total{reporter=\"source\", response_code!~\"5.*\"}[1m])) by (destination_workload, destination_workload_namespace) / sum(rate(istio_requests_total{reporter=\"source\"}[1m])) by (destination_workload, destination_workload_namespace)), \"destination_workload_var\", \".\", \"destination_workload\", \"destination_workload_namespace\")","format":"table","hide":false,"instant":true,"interval":"","intervalFactor":1,"legendFormat":"{{ destination_workload }}.{{ destination_workload_namespace }}","refId":"E"}],"timeFrom":null,"title":"HTTP/GRPC Workloads","transform":"table","type":"table"},{"columns":[],"datasource":"Prometheus","fontSize":"100%","gridPos":{"h":18,"w":24,"x":0,"y":30},"hideTimeOverride":false,"id":109,"links":[],"pageSize":null,"repeatDirection":"v","scroll":true,"showHeader":true,"sort":{"col":5,"desc":true},"styles":[{"alias":"Workload","colorMode":null,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"dateFormat":"YYYY-MM-DD HH:mm:ss","decimals":2,"link":false,"linkTargetBlank":false,"linkTooltip":"$__cell dashboard","linkUrl":"/dashboard/db/istio-workload-dashboard?var-namespace=${__cell_3:raw}&var-workload=${__cell_2:raw}","pattern":"destination_workload","preserveFormat":false,"sanitize":false,"thresholds":[],"type":"hidden","unit":"short"},{"alias":"Bytes Sent","colorMode":null,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"dateFormat":"YYYY-MM-DD HH:mm:ss","decimals":2,"pattern":"Value #A","thresholds":[""],"type":"number","unit":"Bps"},{"alias":"Bytes Received","colorMode":null,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"dateFormat":"YYYY-MM-DD HH:mm:ss","decimals":2,"pattern":"Value #B","thresholds":[],"type":"number","unit":"Bps"},{"alias":"","colorMode":null,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"dateFormat":"YYYY-MM-DD HH:mm:ss","decimals":2,"pattern":"Time","thresholds":[],"type":"hidden","unit":"short"},{"alias":"Workload","colorMode":null,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"dateFormat":"YYYY-MM-DD HH:mm:ss","decimals":2,"link":true,"linkTooltip":"$__cell dashboard","linkUrl":"/dashboard/db/istio-workload-dashboard?var-namespace=${__cell_3:raw}&var-workload=${__cell_2:raw}","pattern":"destination_workload_var","thresholds":[],"type":"string","unit":"short"},{"alias":"","colorMode":null,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"dateFormat":"YYYY-MM-DD HH:mm:ss","decimals":2,"pattern":"destination_workload_namespace","thresholds":[],"type":"hidden","unit":"short"},{"alias":"Service","colorMode":null,"colors":["rgba(245, 54, 54, 0.9)","rgba(237, 129, 40, 0.89)","rgba(50, 172, 45, 0.97)"],"dateFormat":"YYYY-MM-DD HH:mm:ss","decimals":2,"link":true,"linkTooltip":"$__cell dashboard","linkUrl":"/dashboard/db/istio-service-dashboard?var-service=${__cell_1:raw}","pattern":"destination_service","thresholds":[],"type":"number","unit":"short"}],"targets":[{"expr":"label_join(sum(rate(istio_tcp_received_bytes_total{reporter=\"source\"}[1m])) by (destination_workload, destination_workload_namespace, destination_service), \"destination_workload_var\", \".\", \"destination_workload\", \"destination_workload_namespace\")","format":"table","hide":false,"instant":true,"intervalFactor":1,"legendFormat":"{{ destination_workload }}","refId":"A"},{"expr":"label_join(sum(rate(istio_tcp_sent_bytes_total{reporter=\"source\"}[1m])) by (destination_workload, destination_workload_namespace, destination_service), \"destination_workload_var\", \".\", \"destination_workload\", \"destination_workload_namespace\")","format":"table","hide":false,"instant":true,"intervalFactor":1,"legendFormat":"{{ destination_workload }}","refId":"B"}],"timeFrom":null,"title":"TCP Workloads","transform":"table","type":"table"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":9,"w":24,"x":0,"y":48},"id":111,"legend":{"alignAsTable":false,"avg":false,"current":false,"max":false,"min":false,"rightSide":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(istio_build) by (component, tag)","format":"time_series","intervalFactor":1,"legendFormat":"{{ component }}: {{ tag }}","refId":"A"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Istio Components by Version","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}}],"refresh":"5s","schemaVersion":18,"style":"dark","tags":[],"templating":{"list":[{"current":{"selected":true,"text":"default","value":"default"},"hide":0,"includeAll":false,"label":null,"multi":false,"name":"datasource","options":[],"query":"prometheus","queryValue":"","refresh":1,"regex":"","skipUrlSync":false,"type":"datasource"}]},"time":{"from":"now-5m","to":"now"},"timepicker":{"refresh_intervals":["5s","10s","30s","1m","5m","15m","30m","1h","2h","1d"],"time_options":["5m","15m","1h","6h","12h","24h","2d","7d","30d"]},"timezone":"browser","title":"Istio Mesh Dashboard","uid":"G8wLrJIZk","version":5} + istio-service-dashboard.json: "{\"annotations\":{\"list\":[{\"builtIn\":1,\"datasource\":\"-- + Grafana --\",\"enable\":true,\"hide\":true,\"iconColor\":\"rgba(0, 211, 255, 1)\",\"name\":\"Annotations + & Alerts\",\"type\":\"dashboard\"}]},\"editable\":false,\"gnetId\":null,\"graphTooltip\":0,\"iteration\":1595591291797,\"links\":[],\"panels\":[{\"collapsed\":true,\"gridPos\":{\"h\":1,\"w\":24,\"x\":0,\"y\":0},\"id\":106,\"panels\":[{\"content\":\"
\\nSERVICE: $service\\n
\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"gridPos\":{\"h\":3,\"w\":24,\"x\":0,\"y\":1},\"id\":89,\"links\":[],\"mode\":\"html\",\"options\":{\"content\":\"
\\nSERVICE: $service\\n
\",\"mode\":\"html\"},\"pluginVersion\":\"7.1.0\",\"title\":\"\",\"transparent\":true,\"type\":\"text\"},{\"cacheTimeout\":null,\"colorBackground\":false,\"colorValue\":false,\"colors\":[\"rgba(245, + 54, 54, 0.9)\",\"rgba(237, 129, 40, 0.89)\",\"rgba(50, 172, 45, 0.97)\"],\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"format\":\"ops\",\"gauge\":{\"maxValue\":100,\"minValue\":0,\"show\":false,\"thresholdLabels\":false,\"thresholdMarkers\":true},\"gridPos\":{\"h\":4,\"w\":6,\"x\":0,\"y\":4},\"id\":12,\"interval\":null,\"links\":[],\"options\":{\"colorMode\":\"value\",\"graphMode\":\"area\",\"justifyMode\":\"auto\",\"orientation\":\"horizontal\",\"reduceOptions\":{\"calcs\":[\"lastNotNull\"],\"fields\":\"\",\"values\":false},\"textMode\":\"auto\"},\"mappingType\":1,\"mappingTypes\":[{\"name\":\"value + to text\",\"value\":1},{\"name\":\"range to text\",\"value\":2}],\"maxDataPoints\":100,\"nullPointMode\":\"connected\",\"nullText\":null,\"postfix\":\"\",\"postfixFontSize\":\"50%\",\"prefix\":\"\",\"prefixFontSize\":\"50%\",\"rangeMaps\":[{\"from\":\"null\",\"text\":\"N/A\",\"to\":\"null\"}],\"sparkline\":{\"fillColor\":\"rgba(31, + 118, 189, 0.18)\",\"full\":true,\"lineColor\":\"rgb(31, 120, 193)\",\"show\":true},\"tableColumn\":\"\",\"targets\":[{\"expr\":\"round(sum(irate(istio_requests_total{reporter=~\\\"$qrep\\\",destination_service=~\\\"$service\\\"}[5m])), + 0.001)\",\"format\":\"time_series\",\"intervalFactor\":1,\"refId\":\"A\",\"step\":4}],\"thresholds\":\"\",\"title\":\"Client + Request Volume\",\"type\":\"singlestat\",\"valueFontSize\":\"80%\",\"valueMaps\":[{\"op\":\"=\",\"text\":\"N/A\",\"value\":\"null\"}],\"valueName\":\"current\"},{\"cacheTimeout\":null,\"colorBackground\":false,\"colorValue\":false,\"colors\":[\"rgba(50, + 172, 45, 0.97)\",\"rgba(237, 129, 40, 0.89)\",\"rgba(245, 54, 54, 0.9)\"],\"datasource\":\"Prometheus\",\"decimals\":null,\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"format\":\"percentunit\",\"gauge\":{\"maxValue\":100,\"minValue\":80,\"show\":false,\"thresholdLabels\":false,\"thresholdMarkers\":false},\"gridPos\":{\"h\":4,\"w\":6,\"x\":6,\"y\":4},\"id\":14,\"interval\":null,\"links\":[],\"options\":{\"colorMode\":\"value\",\"graphMode\":\"area\",\"justifyMode\":\"auto\",\"orientation\":\"horizontal\",\"reduceOptions\":{\"calcs\":[\"lastNotNull\"],\"fields\":\"\",\"values\":false},\"textMode\":\"auto\"},\"mappingType\":1,\"mappingTypes\":[{\"name\":\"value + to text\",\"value\":1},{\"name\":\"range to text\",\"value\":2}],\"maxDataPoints\":100,\"nullPointMode\":\"connected\",\"nullText\":null,\"postfix\":\"\",\"postfixFontSize\":\"50%\",\"prefix\":\"\",\"prefixFontSize\":\"50%\",\"rangeMaps\":[{\"from\":\"null\",\"text\":\"N/A\",\"to\":\"null\"}],\"sparkline\":{\"fillColor\":\"rgba(31, + 118, 189, 0.18)\",\"full\":true,\"lineColor\":\"rgb(31, 120, 193)\",\"show\":true},\"tableColumn\":\"\",\"targets\":[{\"expr\":\"sum(irate(istio_requests_total{reporter=~\\\"$qrep\\\",destination_service=~\\\"$service\\\",response_code!~\\\"5.*\\\"}[5m])) + / sum(irate(istio_requests_total{reporter=~\\\"$qrep\\\",destination_service=~\\\"$service\\\"}[5m]))\",\"format\":\"time_series\",\"intervalFactor\":1,\"refId\":\"A\"}],\"thresholds\":\"95, + 99, 99.5\",\"title\":\"Client Success Rate (non-5xx responses)\",\"type\":\"singlestat\",\"valueFontSize\":\"80%\",\"valueMaps\":[{\"op\":\"=\",\"text\":\"N/A\",\"value\":\"null\"}],\"valueName\":\"avg\"},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":4,\"w\":6,\"x\":12,\"y\":4},\"hiddenSeries\":false,\"id\":87,\"legend\":{\"alignAsTable\":false,\"avg\":false,\"current\":false,\"hideEmpty\":false,\"hideZero\":false,\"max\":false,\"min\":false,\"rightSide\":true,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"(histogram_quantile(0.50, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=~\\\"$qrep\\\",destination_service=~\\\"$service\\\"}[1m])) + by (le)) / 1000) or histogram_quantile(0.50, sum(irate(istio_request_duration_seconds_bucket{reporter=~\\\"$qrep\\\",destination_service=~\\\"$service\\\"}[1m])) + by (le))\",\"format\":\"time_series\",\"interval\":\"\",\"intervalFactor\":1,\"legendFormat\":\"P50\",\"refId\":\"A\"},{\"expr\":\"(histogram_quantile(0.90, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=~\\\"$qrep\\\",destination_service=~\\\"$service\\\"}[1m])) + by (le)) / 1000) or histogram_quantile(0.90, sum(irate(istio_request_duration_seconds_bucket{reporter=~\\\"$qrep\\\",destination_service=~\\\"$service\\\"}[1m])) + by (le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"P90\",\"refId\":\"B\"},{\"expr\":\"(histogram_quantile(0.99, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=~\\\"$qrep\\\",destination_service=~\\\"$service\\\"}[1m])) + by (le)) / 1000) or histogram_quantile(0.99, sum(irate(istio_request_duration_seconds_bucket{reporter=~\\\"$qrep\\\",destination_service=~\\\"$service\\\"}[1m])) + by (le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"P99\",\"refId\":\"C\"}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Client + Request Duration\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"s\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":false}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"cacheTimeout\":null,\"colorBackground\":false,\"colorValue\":false,\"colors\":[\"#299c46\",\"rgba(237, + 129, 40, 0.89)\",\"#d44a3a\"],\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"format\":\"Bps\",\"gauge\":{\"maxValue\":100,\"minValue\":0,\"show\":false,\"thresholdLabels\":false,\"thresholdMarkers\":true},\"gridPos\":{\"h\":4,\"w\":6,\"x\":18,\"y\":4},\"id\":84,\"interval\":null,\"links\":[],\"options\":{\"colorMode\":\"value\",\"graphMode\":\"area\",\"justifyMode\":\"auto\",\"orientation\":\"horizontal\",\"reduceOptions\":{\"calcs\":[\"lastNotNull\"],\"fields\":\"\",\"values\":false},\"textMode\":\"auto\"},\"mappingType\":1,\"mappingTypes\":[{\"name\":\"value + to text\",\"value\":1},{\"name\":\"range to text\",\"value\":2}],\"maxDataPoints\":100,\"nullPointMode\":\"connected\",\"nullText\":null,\"postfix\":\"\",\"postfixFontSize\":\"50%\",\"prefix\":\"\",\"prefixFontSize\":\"50%\",\"rangeMaps\":[{\"from\":\"null\",\"text\":\"N/A\",\"to\":\"null\"}],\"sparkline\":{\"fillColor\":\"rgba(31, + 118, 189, 0.18)\",\"full\":true,\"lineColor\":\"rgb(31, 120, 193)\",\"show\":true},\"tableColumn\":\"\",\"targets\":[{\"expr\":\"sum(irate(istio_tcp_received_bytes_total{reporter=~\\\"$qrep\\\", + destination_service=~\\\"$service\\\"}[1m]))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"\",\"refId\":\"A\"}],\"thresholds\":\"\",\"title\":\"TCP + Received Bytes\",\"type\":\"singlestat\",\"valueFontSize\":\"80%\",\"valueMaps\":[{\"op\":\"=\",\"text\":\"N/A\",\"value\":\"null\"}],\"valueName\":\"avg\"},{\"cacheTimeout\":null,\"colorBackground\":false,\"colorValue\":false,\"colors\":[\"rgba(245, + 54, 54, 0.9)\",\"rgba(237, 129, 40, 0.89)\",\"rgba(50, 172, 45, 0.97)\"],\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"format\":\"ops\",\"gauge\":{\"maxValue\":100,\"minValue\":0,\"show\":false,\"thresholdLabels\":false,\"thresholdMarkers\":true},\"gridPos\":{\"h\":4,\"w\":6,\"x\":0,\"y\":8},\"id\":97,\"interval\":null,\"links\":[],\"options\":{\"colorMode\":\"value\",\"graphMode\":\"area\",\"justifyMode\":\"auto\",\"orientation\":\"horizontal\",\"reduceOptions\":{\"calcs\":[\"lastNotNull\"],\"fields\":\"\",\"values\":false},\"textMode\":\"auto\"},\"mappingType\":1,\"mappingTypes\":[{\"name\":\"value + to text\",\"value\":1},{\"name\":\"range to text\",\"value\":2}],\"maxDataPoints\":100,\"nullPointMode\":\"connected\",\"nullText\":null,\"postfix\":\"\",\"postfixFontSize\":\"50%\",\"prefix\":\"\",\"prefixFontSize\":\"50%\",\"rangeMaps\":[{\"from\":\"null\",\"text\":\"N/A\",\"to\":\"null\"}],\"sparkline\":{\"fillColor\":\"rgba(31, + 118, 189, 0.18)\",\"full\":true,\"lineColor\":\"rgb(31, 120, 193)\",\"show\":true},\"tableColumn\":\"\",\"targets\":[{\"expr\":\"round(sum(irate(istio_requests_total{reporter=\\\"destination\\\",destination_service=~\\\"$service\\\"}[5m])), + 0.001)\",\"format\":\"time_series\",\"intervalFactor\":1,\"refId\":\"A\",\"step\":4}],\"thresholds\":\"\",\"title\":\"Server + Request Volume\",\"type\":\"singlestat\",\"valueFontSize\":\"80%\",\"valueMaps\":[{\"op\":\"=\",\"text\":\"N/A\",\"value\":\"null\"}],\"valueName\":\"current\"},{\"cacheTimeout\":null,\"colorBackground\":false,\"colorValue\":false,\"colors\":[\"rgba(50, + 172, 45, 0.97)\",\"rgba(237, 129, 40, 0.89)\",\"rgba(245, 54, 54, 0.9)\"],\"datasource\":\"Prometheus\",\"decimals\":null,\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"format\":\"percentunit\",\"gauge\":{\"maxValue\":100,\"minValue\":80,\"show\":false,\"thresholdLabels\":false,\"thresholdMarkers\":false},\"gridPos\":{\"h\":4,\"w\":6,\"x\":6,\"y\":8},\"id\":98,\"interval\":null,\"links\":[],\"options\":{\"colorMode\":\"value\",\"graphMode\":\"area\",\"justifyMode\":\"auto\",\"orientation\":\"horizontal\",\"reduceOptions\":{\"calcs\":[\"lastNotNull\"],\"fields\":\"\",\"values\":false},\"textMode\":\"auto\"},\"mappingType\":1,\"mappingTypes\":[{\"name\":\"value + to text\",\"value\":1},{\"name\":\"range to text\",\"value\":2}],\"maxDataPoints\":100,\"nullPointMode\":\"connected\",\"nullText\":null,\"postfix\":\"\",\"postfixFontSize\":\"50%\",\"prefix\":\"\",\"prefixFontSize\":\"50%\",\"rangeMaps\":[{\"from\":\"null\",\"text\":\"N/A\",\"to\":\"null\"}],\"sparkline\":{\"fillColor\":\"rgba(31, + 118, 189, 0.18)\",\"full\":true,\"lineColor\":\"rgb(31, 120, 193)\",\"show\":true},\"tableColumn\":\"\",\"targets\":[{\"expr\":\"sum(irate(istio_requests_total{reporter=\\\"destination\\\",destination_service=~\\\"$service\\\",response_code!~\\\"5.*\\\"}[5m])) + / sum(irate(istio_requests_total{reporter=\\\"destination\\\",destination_service=~\\\"$service\\\"}[5m]))\",\"format\":\"time_series\",\"intervalFactor\":1,\"refId\":\"A\"}],\"thresholds\":\"95, + 99, 99.5\",\"title\":\"Server Success Rate (non-5xx responses)\",\"type\":\"singlestat\",\"valueFontSize\":\"80%\",\"valueMaps\":[{\"op\":\"=\",\"text\":\"N/A\",\"value\":\"null\"}],\"valueName\":\"avg\"},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":4,\"w\":6,\"x\":12,\"y\":8},\"hiddenSeries\":false,\"id\":99,\"legend\":{\"alignAsTable\":false,\"avg\":false,\"current\":false,\"hideEmpty\":false,\"hideZero\":false,\"max\":false,\"min\":false,\"rightSide\":true,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"(histogram_quantile(0.50, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=\\\"destination\\\",destination_service=~\\\"$service\\\"}[1m])) + by (le)) / 1000) or histogram_quantile(0.50, sum(irate(istio_request_duration_seconds_bucket{reporter=\\\"destination\\\",destination_service=~\\\"$service\\\"}[1m])) + by (le))\",\"format\":\"time_series\",\"interval\":\"\",\"intervalFactor\":1,\"legendFormat\":\"P50\",\"refId\":\"A\"},{\"expr\":\"(histogram_quantile(0.90, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=\\\"destination\\\",destination_service=~\\\"$service\\\"}[1m])) + by (le)) / 1000) or histogram_quantile(0.90, sum(irate(istio_request_duration_seconds_bucket{reporter=\\\"destination\\\",destination_service=~\\\"$service\\\"}[1m])) + by (le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"P90\",\"refId\":\"B\"},{\"expr\":\"(histogram_quantile(0.99, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=\\\"destination\\\",destination_service=~\\\"$service\\\"}[1m])) + by (le)) / 1000) or histogram_quantile(0.99, sum(irate(istio_request_duration_seconds_bucket{reporter=\\\"destination\\\",destination_service=~\\\"$service\\\"}[1m])) + by (le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"P99\",\"refId\":\"C\"}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Server + Request Duration\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"s\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":false}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"cacheTimeout\":null,\"colorBackground\":false,\"colorValue\":false,\"colors\":[\"#299c46\",\"rgba(237, + 129, 40, 0.89)\",\"#d44a3a\"],\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"format\":\"Bps\",\"gauge\":{\"maxValue\":100,\"minValue\":0,\"show\":false,\"thresholdLabels\":false,\"thresholdMarkers\":true},\"gridPos\":{\"h\":4,\"w\":6,\"x\":18,\"y\":8},\"id\":100,\"interval\":null,\"links\":[],\"options\":{\"colorMode\":\"value\",\"graphMode\":\"area\",\"justifyMode\":\"auto\",\"orientation\":\"horizontal\",\"reduceOptions\":{\"calcs\":[\"lastNotNull\"],\"fields\":\"\",\"values\":false},\"textMode\":\"auto\"},\"mappingType\":1,\"mappingTypes\":[{\"name\":\"value + to text\",\"value\":1},{\"name\":\"range to text\",\"value\":2}],\"maxDataPoints\":100,\"nullPointMode\":\"connected\",\"nullText\":null,\"postfix\":\"\",\"postfixFontSize\":\"50%\",\"prefix\":\"\",\"prefixFontSize\":\"50%\",\"rangeMaps\":[{\"from\":\"null\",\"text\":\"N/A\",\"to\":\"null\"}],\"sparkline\":{\"fillColor\":\"rgba(31, + 118, 189, 0.18)\",\"full\":true,\"lineColor\":\"rgb(31, 120, 193)\",\"show\":true},\"tableColumn\":\"\",\"targets\":[{\"expr\":\"sum(irate(istio_tcp_sent_bytes_total{reporter=~\\\"$qrep\\\", + destination_service=~\\\"$service\\\"}[1m]))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"\",\"refId\":\"A\"}],\"thresholds\":\"\",\"title\":\"TCP + Sent Bytes\",\"type\":\"singlestat\",\"valueFontSize\":\"80%\",\"valueMaps\":[{\"op\":\"=\",\"text\":\"N/A\",\"value\":\"null\"}],\"valueName\":\"avg\"}],\"title\":\"General\",\"type\":\"row\"},{\"collapsed\":true,\"gridPos\":{\"h\":1,\"w\":24,\"x\":0,\"y\":1},\"id\":104,\"panels\":[{\"content\":\"
\\nCLIENT WORKLOADS\\n
\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"gridPos\":{\"h\":3,\"w\":24,\"x\":0,\"y\":2},\"id\":45,\"links\":[],\"mode\":\"html\",\"options\":{\"content\":\"
\\nCLIENT WORKLOADS\\n
\",\"mode\":\"html\"},\"pluginVersion\":\"7.1.0\",\"title\":\"\",\"transparent\":true,\"type\":\"text\"},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":0,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":12,\"x\":0,\"y\":5},\"hiddenSeries\":false,\"id\":25,\"legend\":{\"avg\":false,\"current\":false,\"hideEmpty\":true,\"max\":false,\"min\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null + as zero\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"round(sum(irate(istio_requests_total{connection_security_policy=\\\"mutual_tls\\\",destination_service=~\\\"$service\\\",reporter=~\\\"$qrep\\\",source_workload=~\\\"$srcwl\\\",source_workload_namespace=~\\\"$srcns\\\"}[5m])) + by (source_workload, source_workload_namespace, response_code), 0.001)\",\"format\":\"time_series\",\"intervalFactor\":1,\"legendFormat\":\"{{ + source_workload }}.{{ source_workload_namespace }} : {{ response_code }} (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"round(sum(irate(istio_requests_total{connection_security_policy!=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", reporter=~\\\"$qrep\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[5m])) by (source_workload, source_workload_namespace, + response_code), 0.001)\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + source_workload }}.{{ source_workload_namespace }} : {{ response_code }}\",\"refId\":\"B\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Incoming + Requests By Source And Response Code\",\"tooltip\":{\"shared\":false,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[\"total\"]},\"yaxes\":[{\"format\":\"ops\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":false}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":12,\"x\":12,\"y\":5},\"hiddenSeries\":false,\"id\":26,\"legend\":{\"avg\":false,\"current\":false,\"hideEmpty\":true,\"hideZero\":false,\"max\":false,\"min\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"sum(irate(istio_requests_total{reporter=~\\\"$qrep\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\",response_code!~\\\"5.*\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[5m])) + by (source_workload, source_workload_namespace) / sum(irate(istio_requests_total{reporter=~\\\"$qrep\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[5m])) + by (source_workload, source_workload_namespace)\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + source_workload }}.{{ source_workload_namespace }} (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"sum(irate(istio_requests_total{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\",response_code!~\\\"5.*\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[5m])) + by (source_workload, source_workload_namespace) / sum(irate(istio_requests_total{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[5m])) + by (source_workload, source_workload_namespace)\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + source_workload }}.{{ source_workload_namespace }}\",\"refId\":\"B\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Incoming + Success Rate (non-5xx responses) By Source\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"percentunit\",\"label\":null,\"logBase\":1,\"max\":\"1.01\",\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":false}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"description\":\"\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":8,\"x\":0,\"y\":11},\"hiddenSeries\":false,\"id\":27,\"legend\":{\"alignAsTable\":false,\"avg\":false,\"current\":false,\"hideEmpty\":true,\"hideZero\":false,\"max\":false,\"min\":false,\"rightSide\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"(histogram_quantile(0.50, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le)) / 1000) or histogram_quantile(0.50, + sum(irate(istio_request_duration_seconds_bucket{reporter=~\\\"$qrep\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P50 (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"(histogram_quantile(0.90, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le)) / 1000) or histogram_quantile(0.90, + sum(irate(istio_request_duration_seconds_bucket{reporter=~\\\"$qrep\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P90 (\U0001F510mTLS)\",\"refId\":\"B\",\"step\":2},{\"expr\":\"(histogram_quantile(0.95, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le)) / 1000) or histogram_quantile(0.95, + sum(irate(istio_request_duration_seconds_bucket{reporter=~\\\"$qrep\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P95 (\U0001F510mTLS)\",\"refId\":\"C\",\"step\":2},{\"expr\":\"(histogram_quantile(0.99, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le)) / 1000) or histogram_quantile(0.99, + sum(irate(istio_request_duration_seconds_bucket{reporter=~\\\"$qrep\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P99 (\U0001F510mTLS)\",\"refId\":\"D\",\"step\":2},{\"expr\":\"(histogram_quantile(0.50, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le)) / 1000) or histogram_quantile(0.50, + sum(irate(istio_request_duration_seconds_bucket{reporter=~\\\"$qrep\\\", connection_security_policy!=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P50\",\"refId\":\"E\",\"step\":2},{\"expr\":\"(histogram_quantile(0.90, sum(irate(istio_request_duration_milliseconds_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le)) / 1000) or histogram_quantile(0.90, + sum(irate(istio_request_duration_seconds_bucket{reporter=~\\\"$qrep\\\", connection_security_policy!=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P90\",\"refId\":\"F\",\"step\":2},{\"expr\":\"(histogram_quantile(0.95, sum(irate(istio_request_duration_milliseconds_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le)) / 1000) or histogram_quantile(0.95, + sum(irate(istio_request_duration_seconds_bucket{reporter=~\\\"$qrep\\\", connection_security_policy!=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P95\",\"refId\":\"G\",\"step\":2},{\"expr\":\"(histogram_quantile(0.99, sum(irate(istio_request_duration_milliseconds_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le)) / 1000) or histogram_quantile(0.99, + sum(irate(istio_request_duration_seconds_bucket{reporter=~\\\"$qrep\\\", connection_security_policy!=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P99\",\"refId\":\"H\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Incoming + Request Duration By Source\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"s\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":false}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":8,\"x\":8,\"y\":11},\"hiddenSeries\":false,\"id\":28,\"legend\":{\"alignAsTable\":false,\"avg\":false,\"current\":false,\"hideEmpty\":true,\"max\":false,\"min\":false,\"rightSide\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"histogram_quantile(0.50, + sum(irate(istio_request_bytes_bucket{reporter=~\\\"$qrep\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P50 (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"histogram_quantile(0.90, + sum(irate(istio_request_bytes_bucket{reporter=~\\\"$qrep\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + \ P90 (\U0001F510mTLS)\",\"refId\":\"B\",\"step\":2},{\"expr\":\"histogram_quantile(0.95, + sum(irate(istio_request_bytes_bucket{reporter=~\\\"$qrep\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P95 (\U0001F510mTLS)\",\"refId\":\"C\",\"step\":2},{\"expr\":\"histogram_quantile(0.99, + sum(irate(istio_request_bytes_bucket{reporter=~\\\"$qrep\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + \ P99 (\U0001F510mTLS)\",\"refId\":\"D\",\"step\":2},{\"expr\":\"histogram_quantile(0.50, + sum(irate(istio_request_bytes_bucket{reporter=~\\\"$qrep\\\", connection_security_policy!=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P50\",\"refId\":\"E\",\"step\":2},{\"expr\":\"histogram_quantile(0.90, sum(irate(istio_request_bytes_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P90\",\"refId\":\"F\",\"step\":2},{\"expr\":\"histogram_quantile(0.95, sum(irate(istio_request_bytes_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P95\",\"refId\":\"G\",\"step\":2},{\"expr\":\"histogram_quantile(0.99, sum(irate(istio_request_bytes_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P99\",\"refId\":\"H\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Incoming + Request Size By Source\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"decbytes\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":false}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":8,\"x\":16,\"y\":11},\"hiddenSeries\":false,\"id\":68,\"legend\":{\"alignAsTable\":false,\"avg\":false,\"current\":false,\"hideEmpty\":true,\"max\":false,\"min\":false,\"rightSide\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"histogram_quantile(0.50, + sum(irate(istio_response_bytes_bucket{reporter=~\\\"$qrep\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P50 (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"histogram_quantile(0.90, + sum(irate(istio_response_bytes_bucket{reporter=~\\\"$qrep\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + \ P90 (\U0001F510mTLS)\",\"refId\":\"B\",\"step\":2},{\"expr\":\"histogram_quantile(0.95, + sum(irate(istio_response_bytes_bucket{reporter=~\\\"$qrep\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P95 (\U0001F510mTLS)\",\"refId\":\"C\",\"step\":2},{\"expr\":\"histogram_quantile(0.99, + sum(irate(istio_response_bytes_bucket{reporter=~\\\"$qrep\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + \ P99 (\U0001F510mTLS)\",\"refId\":\"D\",\"step\":2},{\"expr\":\"histogram_quantile(0.50, + sum(irate(istio_response_bytes_bucket{reporter=~\\\"$qrep\\\", connection_security_policy!=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P50\",\"refId\":\"E\",\"step\":2},{\"expr\":\"histogram_quantile(0.90, sum(irate(istio_response_bytes_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P90\",\"refId\":\"F\",\"step\":2},{\"expr\":\"histogram_quantile(0.95, sum(irate(istio_response_bytes_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P95\",\"refId\":\"G\",\"step\":2},{\"expr\":\"histogram_quantile(0.99, sum(irate(istio_response_bytes_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P99\",\"refId\":\"H\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Response + Size By Source\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"decbytes\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":false}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":12,\"x\":0,\"y\":17},\"hiddenSeries\":false,\"id\":80,\"legend\":{\"avg\":false,\"current\":false,\"max\":false,\"min\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"round(sum(irate(istio_tcp_received_bytes_total{reporter=~\\\"$qrep\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace), 0.001)\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + source_workload }}.{{ source_workload_namespace}} (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"round(sum(irate(istio_tcp_received_bytes_total{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace), 0.001)\",\"format\":\"time_series\",\"intervalFactor\":1,\"legendFormat\":\"{{ + source_workload }}.{{ source_workload_namespace}}\",\"refId\":\"B\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Bytes + Received from Incoming TCP Connection\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"Bps\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":true}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":12,\"x\":12,\"y\":17},\"hiddenSeries\":false,\"id\":82,\"legend\":{\"avg\":false,\"current\":false,\"max\":false,\"min\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"round(sum(irate(istio_tcp_sent_bytes_total{connection_security_policy=\\\"mutual_tls\\\", + reporter=~\\\"$qrep\\\", destination_service=~\\\"$service\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[1m])) by (source_workload, source_workload_namespace), + 0.001)\",\"format\":\"time_series\",\"intervalFactor\":1,\"legendFormat\":\"{{ + source_workload }}.{{ source_workload_namespace}} (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"round(sum(irate(istio_tcp_sent_bytes_total{connection_security_policy!=\\\"mutual_tls\\\", + reporter=~\\\"$qrep\\\", destination_service=~\\\"$service\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[1m])) by (source_workload, source_workload_namespace), + 0.001)\",\"format\":\"time_series\",\"intervalFactor\":1,\"legendFormat\":\"{{ + source_workload }}.{{ source_workload_namespace}}\",\"refId\":\"B\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Bytes + Sent to Incoming TCP Connection\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"Bps\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":true}],\"yaxis\":{\"align\":false,\"alignLevel\":null}}],\"title\":\"Client + Workloads\",\"type\":\"row\"},{\"collapsed\":true,\"gridPos\":{\"h\":1,\"w\":24,\"x\":0,\"y\":2},\"id\":102,\"panels\":[{\"content\":\"
\\nSERVICE WORKLOADS\\n
\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"gridPos\":{\"h\":3,\"w\":24,\"x\":0,\"y\":3},\"id\":69,\"links\":[],\"mode\":\"html\",\"options\":{\"content\":\"
\\nSERVICE WORKLOADS\\n
\",\"mode\":\"html\"},\"pluginVersion\":\"7.1.0\",\"title\":\"\",\"transparent\":true,\"type\":\"text\"},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":0,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":12,\"x\":0,\"y\":6},\"hiddenSeries\":false,\"id\":90,\"legend\":{\"avg\":false,\"current\":false,\"hideEmpty\":true,\"max\":false,\"min\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null + as zero\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"round(sum(irate(istio_requests_total{connection_security_policy=\\\"mutual_tls\\\",destination_service=~\\\"$service\\\",reporter=\\\"destination\\\",destination_workload=~\\\"$dstwl\\\",destination_workload_namespace=~\\\"$dstns\\\"}[5m])) + by (destination_workload, destination_workload_namespace, response_code), 0.001)\",\"format\":\"time_series\",\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} : {{ response_code + }} (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"round(sum(irate(istio_requests_total{connection_security_policy!=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", reporter=\\\"destination\\\", destination_workload=~\\\"$dstwl\\\", + destination_workload_namespace=~\\\"$dstns\\\"}[5m])) by (destination_workload, + destination_workload_namespace, response_code), 0.001)\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} : {{ response_code + }}\",\"refId\":\"B\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Incoming + Requests By Destination Workload And Response Code\",\"tooltip\":{\"shared\":false,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[\"total\"]},\"yaxes\":[{\"format\":\"ops\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":false}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":12,\"x\":12,\"y\":6},\"hiddenSeries\":false,\"id\":91,\"legend\":{\"avg\":false,\"current\":false,\"hideEmpty\":true,\"hideZero\":false,\"max\":false,\"min\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"sum(irate(istio_requests_total{reporter=\\\"destination\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\",response_code!~\\\"5.*\\\", + destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[5m])) + by (destination_workload, destination_workload_namespace) / sum(irate(istio_requests_total{reporter=\\\"destination\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[5m])) + by (destination_workload, destination_workload_namespace)\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"sum(irate(istio_requests_total{reporter=\\\"destination\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\",response_code!~\\\"5.*\\\", + destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[5m])) + by (destination_workload, destination_workload_namespace) / sum(irate(istio_requests_total{reporter=\\\"destination\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[5m])) + by (destination_workload, destination_workload_namespace)\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }}\",\"refId\":\"B\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Incoming + Success Rate (non-5xx responses) By Destination Workload\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"percentunit\",\"label\":null,\"logBase\":1,\"max\":\"1.01\",\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":false}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"description\":\"\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":8,\"x\":0,\"y\":12},\"hiddenSeries\":false,\"id\":94,\"legend\":{\"alignAsTable\":false,\"avg\":false,\"current\":false,\"hideEmpty\":true,\"hideZero\":false,\"max\":false,\"min\":false,\"rightSide\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"(histogram_quantile(0.50, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=\\\"destination\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le)) / 1000) or histogram_quantile(0.50, + sum(irate(istio_request_duration_seconds_bucket{reporter=\\\"destination\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} P50 (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"(histogram_quantile(0.90, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=\\\"destination\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le)) / 1000) or histogram_quantile(0.90, + sum(irate(istio_request_duration_seconds_bucket{reporter=\\\"destination\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} P90 (\U0001F510mTLS)\",\"refId\":\"B\",\"step\":2},{\"expr\":\"(histogram_quantile(0.95, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=\\\"destination\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le)) / 1000) or histogram_quantile(0.95, + sum(irate(istio_request_duration_seconds_bucket{reporter=\\\"destination\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} P95 (\U0001F510mTLS)\",\"refId\":\"C\",\"step\":2},{\"expr\":\"(histogram_quantile(0.99, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=\\\"destination\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le)) / 1000) or histogram_quantile(0.99, + sum(irate(istio_request_duration_seconds_bucket{reporter=\\\"destination\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} P99 (\U0001F510mTLS)\",\"refId\":\"D\",\"step\":2},{\"expr\":\"(histogram_quantile(0.50, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=\\\"destination\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le)) / 1000) or histogram_quantile(0.50, + sum(irate(istio_request_duration_seconds_bucket{reporter=\\\"destination\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} P50\",\"refId\":\"E\",\"step\":2},{\"expr\":\"(histogram_quantile(0.90, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=\\\"destination\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le)) / 1000) or histogram_quantile(0.90, + sum(irate(istio_request_duration_seconds_bucket{reporter=\\\"destination\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} P90\",\"refId\":\"F\",\"step\":2},{\"expr\":\"(histogram_quantile(0.95, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=\\\"destination\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le)) / 1000) or histogram_quantile(0.95, + sum(irate(istio_request_duration_seconds_bucket{reporter=\\\"destination\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} P95\",\"refId\":\"G\",\"step\":2},{\"expr\":\"(histogram_quantile(0.99, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=\\\"destination\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le)) / 1000) or histogram_quantile(0.99, + sum(irate(istio_request_duration_seconds_bucket{reporter=\\\"destination\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} P99\",\"refId\":\"H\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Incoming + Request Duration By Service Workload\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"s\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":false}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":8,\"x\":8,\"y\":12},\"hiddenSeries\":false,\"id\":95,\"legend\":{\"alignAsTable\":false,\"avg\":false,\"current\":false,\"hideEmpty\":true,\"max\":false,\"min\":false,\"rightSide\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"histogram_quantile(0.50, + sum(irate(istio_request_bytes_bucket{reporter=\\\"destination\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} P50 (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"histogram_quantile(0.90, + sum(irate(istio_request_bytes_bucket{reporter=\\\"destination\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} P90 (\U0001F510mTLS)\",\"refId\":\"B\",\"step\":2},{\"expr\":\"histogram_quantile(0.95, + sum(irate(istio_request_bytes_bucket{reporter=\\\"destination\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} P95 (\U0001F510mTLS)\",\"refId\":\"C\",\"step\":2},{\"expr\":\"histogram_quantile(0.99, + sum(irate(istio_request_bytes_bucket{reporter=\\\"destination\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} P99 (\U0001F510mTLS)\",\"refId\":\"D\",\"step\":2},{\"expr\":\"histogram_quantile(0.50, + sum(irate(istio_request_bytes_bucket{reporter=\\\"destination\\\", connection_security_policy!=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} P50\",\"refId\":\"E\",\"step\":2},{\"expr\":\"histogram_quantile(0.90, + sum(irate(istio_request_bytes_bucket{reporter=\\\"destination\\\", connection_security_policy!=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} P90\",\"refId\":\"F\",\"step\":2},{\"expr\":\"histogram_quantile(0.95, + sum(irate(istio_request_bytes_bucket{reporter=\\\"destination\\\", connection_security_policy!=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} P95\",\"refId\":\"G\",\"step\":2},{\"expr\":\"histogram_quantile(0.99, + sum(irate(istio_request_bytes_bucket{reporter=\\\"destination\\\", connection_security_policy!=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} P99\",\"refId\":\"H\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Incoming + Request Size By Service Workload\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"decbytes\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":false}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":8,\"x\":16,\"y\":12},\"hiddenSeries\":false,\"id\":96,\"legend\":{\"alignAsTable\":false,\"avg\":false,\"current\":false,\"hideEmpty\":true,\"max\":false,\"min\":false,\"rightSide\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"histogram_quantile(0.50, + sum(irate(istio_response_bytes_bucket{reporter=\\\"destination\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} P50 (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"histogram_quantile(0.90, + sum(irate(istio_response_bytes_bucket{reporter=\\\"destination\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} P90 (\U0001F510mTLS)\",\"refId\":\"B\",\"step\":2},{\"expr\":\"histogram_quantile(0.95, + sum(irate(istio_response_bytes_bucket{reporter=\\\"destination\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} P95 (\U0001F510mTLS)\",\"refId\":\"C\",\"step\":2},{\"expr\":\"histogram_quantile(0.99, + sum(irate(istio_response_bytes_bucket{reporter=\\\"destination\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} P99 (\U0001F510mTLS)\",\"refId\":\"D\",\"step\":2},{\"expr\":\"histogram_quantile(0.50, + sum(irate(istio_response_bytes_bucket{reporter=\\\"destination\\\", connection_security_policy!=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} P50\",\"refId\":\"E\",\"step\":2},{\"expr\":\"histogram_quantile(0.90, + sum(irate(istio_response_bytes_bucket{reporter=\\\"destination\\\", connection_security_policy!=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} P90\",\"refId\":\"F\",\"step\":2},{\"expr\":\"histogram_quantile(0.95, + sum(irate(istio_response_bytes_bucket{reporter=\\\"destination\\\", connection_security_policy!=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} P95\",\"refId\":\"G\",\"step\":2},{\"expr\":\"histogram_quantile(0.99, + sum(irate(istio_response_bytes_bucket{reporter=\\\"destination\\\", connection_security_policy!=\\\"mutual_tls\\\", + destination_service=~\\\"$service\\\", destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace }} P99\",\"refId\":\"H\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Response + Size By Service Workload\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"decbytes\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":false}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":12,\"x\":0,\"y\":18},\"hiddenSeries\":false,\"id\":92,\"legend\":{\"avg\":false,\"current\":false,\"max\":false,\"min\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"round(sum(irate(istio_tcp_received_bytes_total{reporter=\\\"destination\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace), 0.001)\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace}} (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"round(sum(irate(istio_tcp_received_bytes_total{reporter=\\\"destination\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_service=~\\\"$service\\\", + destination_workload=~\\\"$dstwl\\\", destination_workload_namespace=~\\\"$dstns\\\"}[1m])) + by (destination_workload, destination_workload_namespace), 0.001)\",\"format\":\"time_series\",\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{ destination_workload_namespace}}\",\"refId\":\"B\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Bytes + Received from Incoming TCP Connection\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"Bps\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":true}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":12,\"x\":12,\"y\":18},\"hiddenSeries\":false,\"id\":93,\"legend\":{\"avg\":false,\"current\":false,\"max\":false,\"min\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"round(sum(irate(istio_tcp_sent_bytes_total{connection_security_policy=\\\"mutual_tls\\\", + reporter=\\\"destination\\\", destination_service=~\\\"$service\\\", destination_workload=~\\\"$dstwl\\\", + destination_workload_namespace=~\\\"$dstns\\\"}[1m])) by (destination_workload, + destination_workload_namespace), 0.001)\",\"format\":\"time_series\",\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{destination_workload_namespace }} (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"round(sum(irate(istio_tcp_sent_bytes_total{connection_security_policy!=\\\"mutual_tls\\\", + reporter=\\\"destination\\\", destination_service=~\\\"$service\\\", destination_workload=~\\\"$dstwl\\\", + destination_workload_namespace=~\\\"$dstns\\\"}[1m])) by (destination_workload, + destination_workload_namespace), 0.001)\",\"format\":\"time_series\",\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_workload }}.{{destination_workload_namespace }}\",\"refId\":\"B\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Bytes + Sent to Incoming TCP Connection\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"Bps\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":true}],\"yaxis\":{\"align\":false,\"alignLevel\":null}}],\"title\":\"Service + Workloads\",\"type\":\"row\"}],\"refresh\":\"1m\",\"schemaVersion\":26,\"style\":\"dark\",\"tags\":[],\"templating\":{\"list\":[{\"current\":{\"selected\":true,\"text\":\"default\",\"value\":\"default\"},\"hide\":0,\"includeAll\":false,\"label\":null,\"multi\":false,\"name\":\"datasource\",\"options\":[],\"query\":\"prometheus\",\"queryValue\":\"\",\"refresh\":1,\"regex\":\"\",\"skipUrlSync\":false,\"type\":\"datasource\"},{\"allValue\":null,\"current\":{},\"datasource\":\"Prometheus\",\"definition\":\"\",\"hide\":0,\"includeAll\":false,\"label\":\"Service\",\"multi\":false,\"name\":\"service\",\"options\":[],\"query\":\"label_values(destination_service)\",\"refresh\":1,\"regex\":\"\",\"skipUrlSync\":false,\"sort\":0,\"tagValuesQuery\":\"\",\"tags\":[],\"tagsQuery\":\"\",\"type\":\"query\",\"useTags\":false},{\"allValue\":null,\"current\":{\"selected\":true,\"text\":\"destination\",\"value\":\"destination\"},\"datasource\":\"Prometheus\",\"definition\":\"\",\"hide\":0,\"includeAll\":false,\"label\":\"Reporter\",\"multi\":true,\"name\":\"qrep\",\"options\":[],\"query\":\"label_values(reporter)\",\"refresh\":1,\"regex\":\"\",\"skipUrlSync\":false,\"sort\":1,\"tagValuesQuery\":\"\",\"tags\":[],\"tagsQuery\":\"\",\"type\":\"query\",\"useTags\":false},{\"allValue\":null,\"current\":{},\"datasource\":\"Prometheus\",\"definition\":\"\",\"hide\":0,\"includeAll\":true,\"label\":\"Client + Cluster\",\"multi\":true,\"name\":\"srccluster\",\"options\":[],\"query\":\"query_result(sum(istio_requests_total{reporter=~\\\"$qrep\\\", + destination_service=\\\"$service\\\"}) by (source_cluster) or sum(istio_tcp_sent_bytes_total{reporter=~\\\"$qrep\\\", + destination_service=~\\\"$service\\\"}) by (source_cluster))\",\"refresh\":1,\"regex\":\"/.*cluster=\\\"([^\\\"]*).*/\",\"skipUrlSync\":false,\"sort\":2,\"tagValuesQuery\":\"\",\"tags\":[],\"tagsQuery\":\"\",\"type\":\"query\",\"useTags\":false},{\"allValue\":null,\"current\":{},\"datasource\":\"Prometheus\",\"definition\":\"\",\"hide\":0,\"includeAll\":true,\"label\":\"Client + Workload Namespace\",\"multi\":true,\"name\":\"srcns\",\"options\":[],\"query\":\"query_result(sum(istio_requests_total{reporter=~\\\"$qrep\\\", + destination_service=\\\"$service\\\"}) by (source_workload_namespace) or sum(istio_tcp_sent_bytes_total{reporter=~\\\"$qrep\\\", + destination_service=~\\\"$service\\\"}) by (source_workload_namespace))\",\"refresh\":1,\"regex\":\"/.*namespace=\\\"([^\\\"]*).*/\",\"skipUrlSync\":false,\"sort\":3,\"tagValuesQuery\":\"\",\"tags\":[],\"tagsQuery\":\"\",\"type\":\"query\",\"useTags\":false},{\"allValue\":null,\"current\":{},\"datasource\":\"Prometheus\",\"definition\":\"\",\"hide\":0,\"includeAll\":true,\"label\":\"Client + Workload\",\"multi\":true,\"name\":\"srcwl\",\"options\":[],\"query\":\"query_result(sum(istio_requests_total{reporter=~\\\"$qrep\\\", + destination_service=~\\\"$service\\\", source_workload_namespace=~\\\"$srcns\\\"}) + by (source_workload) or sum(istio_tcp_sent_bytes_total{reporter=~\\\"$qrep\\\", + destination_service=~\\\"$service\\\", source_workload_namespace=~\\\"$srcns\\\"}) + by (source_workload))\",\"refresh\":1,\"regex\":\"/.*workload=\\\"([^\\\"]*).*/\",\"skipUrlSync\":false,\"sort\":4,\"tagValuesQuery\":\"\",\"tags\":[],\"tagsQuery\":\"\",\"type\":\"query\",\"useTags\":false},{\"allValue\":null,\"current\":{},\"datasource\":\"Prometheus\",\"definition\":\"\",\"hide\":0,\"includeAll\":true,\"label\":\"Service + Workload Cluster\",\"multi\":true,\"name\":\"dstcluster\",\"options\":[],\"query\":\"query_result(sum(istio_requests_total{reporter=\\\"destination\\\", + destination_service=\\\"$service\\\"}) by (destination_cluster) or sum(istio_tcp_sent_bytes_total{reporter=\\\"destination\\\", + destination_service=~\\\"$service\\\"}) by (destination_cluster))\",\"refresh\":1,\"regex\":\"/.*cluster=\\\"([^\\\"]*).*/\",\"skipUrlSync\":false,\"sort\":2,\"tagValuesQuery\":\"\",\"tags\":[],\"tagsQuery\":\"\",\"type\":\"query\",\"useTags\":false},{\"allValue\":null,\"current\":{},\"datasource\":\"Prometheus\",\"definition\":\"\",\"hide\":0,\"includeAll\":true,\"label\":\"Service + Workload Namespace\",\"multi\":true,\"name\":\"dstns\",\"options\":[],\"query\":\"query_result(sum(istio_requests_total{reporter=\\\"destination\\\", + destination_service=\\\"$service\\\"}) by (destination_workload_namespace) or + sum(istio_tcp_sent_bytes_total{reporter=\\\"destination\\\", destination_service=~\\\"$service\\\"}) + by (destination_workload_namespace))\",\"refresh\":1,\"regex\":\"/.*namespace=\\\"([^\\\"]*).*/\",\"skipUrlSync\":false,\"sort\":3,\"tagValuesQuery\":\"\",\"tags\":[],\"tagsQuery\":\"\",\"type\":\"query\",\"useTags\":false},{\"allValue\":null,\"current\":{},\"datasource\":\"Prometheus\",\"definition\":\"\",\"hide\":0,\"includeAll\":true,\"label\":\"Service + Workload\",\"multi\":true,\"name\":\"dstwl\",\"options\":[],\"query\":\"query_result( + sum(istio_requests_total{reporter=\\\"destination\\\", destination_service=~\\\"$service\\\", + destination_cluster=~\\\"$dstcluster\\\", destination_workload_namespace=~\\\"$dstns\\\"}) + by (destination_workload) or sum(istio_tcp_sent_bytes_total{reporter=\\\"destination\\\", + destination_service=~\\\"$service\\\", destination_cluster=~\\\"$dstcluster\\\", + destination_workload_namespace=~\\\"$dstns\\\"}) by (destination_workload))\",\"refresh\":1,\"regex\":\"/.*workload=\\\"([^\\\"]*).*/\",\"skipUrlSync\":false,\"sort\":4,\"tagValuesQuery\":\"\",\"tags\":[],\"tagsQuery\":\"\",\"type\":\"query\",\"useTags\":false}]},\"time\":{\"from\":\"now-5m\",\"to\":\"now\"},\"timepicker\":{\"refresh_intervals\":[\"5m\",\"15m\",\"30m\",\"1h\",\"2h\",\"1d\"],\"time_options\":[\"5m\",\"15m\",\"1h\",\"6h\",\"12h\",\"24h\",\"2d\",\"7d\",\"30d\"]},\"timezone\":\"\",\"title\":\"Istio + Service Dashboard\",\"uid\":\"LJ_uJAvmk\",\"version\":1}\n" + istio-workload-dashboard.json: "{\"annotations\":{\"list\":[{\"builtIn\":1,\"datasource\":\"-- + Grafana --\",\"enable\":true,\"hide\":true,\"iconColor\":\"rgba(0, 211, 255, 1)\",\"name\":\"Annotations + & Alerts\",\"type\":\"dashboard\"}]},\"editable\":false,\"gnetId\":null,\"graphTooltip\":0,\"iteration\":1531345461465,\"links\":[],\"panels\":[{\"collapsed\":true,\"gridPos\":{\"h\":1,\"w\":24,\"x\":0,\"y\":0},\"id\":95,\"panels\":[{\"content\":\"
\\nWORKLOAD: $workload.$namespace\\n
\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"gridPos\":{\"h\":3,\"w\":24,\"x\":0,\"y\":1},\"id\":89,\"links\":[],\"mode\":\"html\",\"options\":{\"content\":\"
\\nWORKLOAD: $workload.$namespace\\n
\",\"mode\":\"html\"},\"pluginVersion\":\"7.1.0\",\"title\":\"\",\"transparent\":true,\"type\":\"text\"},{\"cacheTimeout\":null,\"colorBackground\":false,\"colorValue\":false,\"colors\":[\"rgba(245, + 54, 54, 0.9)\",\"rgba(237, 129, 40, 0.89)\",\"rgba(50, 172, 45, 0.97)\"],\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"format\":\"ops\",\"gauge\":{\"maxValue\":100,\"minValue\":0,\"show\":false,\"thresholdLabels\":false,\"thresholdMarkers\":true},\"gridPos\":{\"h\":4,\"w\":8,\"x\":0,\"y\":4},\"id\":12,\"interval\":null,\"links\":[],\"options\":{\"colorMode\":\"value\",\"graphMode\":\"area\",\"justifyMode\":\"auto\",\"orientation\":\"horizontal\",\"reduceOptions\":{\"calcs\":[\"lastNotNull\"],\"fields\":\"\",\"values\":false},\"textMode\":\"auto\"},\"mappingType\":1,\"mappingTypes\":[{\"name\":\"value + to text\",\"value\":1},{\"name\":\"range to text\",\"value\":2}],\"maxDataPoints\":100,\"nullPointMode\":\"connected\",\"nullText\":null,\"postfix\":\"\",\"postfixFontSize\":\"50%\",\"prefix\":\"\",\"prefixFontSize\":\"50%\",\"rangeMaps\":[{\"from\":\"null\",\"text\":\"N/A\",\"to\":\"null\"}],\"sparkline\":{\"fillColor\":\"rgba(31, + 118, 189, 0.18)\",\"full\":true,\"lineColor\":\"rgb(31, 120, 193)\",\"show\":true},\"tableColumn\":\"\",\"targets\":[{\"expr\":\"round(sum(irate(istio_requests_total{reporter=~\\\"$qrep\\\",destination_workload_namespace=~\\\"$namespace\\\",destination_workload=~\\\"$workload\\\"}[5m])), + 0.001)\",\"format\":\"time_series\",\"intervalFactor\":1,\"refId\":\"A\",\"step\":4}],\"thresholds\":\"\",\"title\":\"Incoming + Request Volume\",\"type\":\"singlestat\",\"valueFontSize\":\"80%\",\"valueMaps\":[{\"op\":\"=\",\"text\":\"N/A\",\"value\":\"null\"}],\"valueName\":\"current\"},{\"cacheTimeout\":null,\"colorBackground\":false,\"colorValue\":false,\"colors\":[\"rgba(50, + 172, 45, 0.97)\",\"rgba(237, 129, 40, 0.89)\",\"rgba(245, 54, 54, 0.9)\"],\"datasource\":\"Prometheus\",\"decimals\":null,\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"format\":\"percentunit\",\"gauge\":{\"maxValue\":100,\"minValue\":80,\"show\":false,\"thresholdLabels\":false,\"thresholdMarkers\":false},\"gridPos\":{\"h\":4,\"w\":8,\"x\":8,\"y\":4},\"id\":14,\"interval\":null,\"links\":[],\"options\":{\"colorMode\":\"value\",\"graphMode\":\"area\",\"justifyMode\":\"auto\",\"orientation\":\"horizontal\",\"reduceOptions\":{\"calcs\":[\"lastNotNull\"],\"fields\":\"\",\"values\":false},\"textMode\":\"auto\"},\"mappingType\":1,\"mappingTypes\":[{\"name\":\"value + to text\",\"value\":1},{\"name\":\"range to text\",\"value\":2}],\"maxDataPoints\":100,\"nullPointMode\":\"connected\",\"nullText\":null,\"postfix\":\"\",\"postfixFontSize\":\"50%\",\"prefix\":\"\",\"prefixFontSize\":\"50%\",\"rangeMaps\":[{\"from\":\"null\",\"text\":\"N/A\",\"to\":\"null\"}],\"sparkline\":{\"fillColor\":\"rgba(31, + 118, 189, 0.18)\",\"full\":true,\"lineColor\":\"rgb(31, 120, 193)\",\"show\":true},\"tableColumn\":\"\",\"targets\":[{\"expr\":\"sum(irate(istio_requests_total{reporter=~\\\"$qrep\\\",destination_workload_namespace=~\\\"$namespace\\\",destination_workload=~\\\"$workload\\\",response_code!~\\\"5.*\\\"}[5m])) + / sum(irate(istio_requests_total{reporter=~\\\"$qrep\\\",destination_workload_namespace=~\\\"$namespace\\\",destination_workload=~\\\"$workload\\\"}[5m]))\",\"format\":\"time_series\",\"intervalFactor\":1,\"refId\":\"A\"}],\"thresholds\":\"95, + 99, 99.5\",\"title\":\"Incoming Success Rate (non-5xx responses)\",\"type\":\"singlestat\",\"valueFontSize\":\"80%\",\"valueMaps\":[{\"op\":\"=\",\"text\":\"N/A\",\"value\":\"null\"}],\"valueName\":\"avg\"},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":4,\"w\":8,\"x\":16,\"y\":4},\"hiddenSeries\":false,\"id\":87,\"legend\":{\"alignAsTable\":false,\"avg\":false,\"current\":false,\"hideEmpty\":false,\"hideZero\":false,\"max\":false,\"min\":false,\"rightSide\":true,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"(histogram_quantile(0.50, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=~\\\"$qrep\\\",destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\"}[1m])) by (le)) / 1000) or + histogram_quantile(0.50, sum(irate(istio_request_duration_seconds_bucket{reporter=~\\\"$qrep\\\",destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\"}[1m])) by (le))\",\"format\":\"time_series\",\"interval\":\"\",\"intervalFactor\":1,\"legendFormat\":\"P50\",\"refId\":\"A\"},{\"expr\":\"(histogram_quantile(0.90, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=~\\\"$qrep\\\",destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\"}[1m])) by (le)) / 1000) or + histogram_quantile(0.90, sum(irate(istio_request_duration_seconds_bucket{reporter=~\\\"$qrep\\\",destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\"}[1m])) by (le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"P90\",\"refId\":\"B\"},{\"expr\":\"(histogram_quantile(0.99, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=~\\\"$qrep\\\",destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\"}[1m])) by (le)) / 1000) or + histogram_quantile(0.99, sum(irate(istio_request_duration_seconds_bucket{reporter=~\\\"$qrep\\\",destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\"}[1m])) by (le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"P99\",\"refId\":\"C\"}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Request + Duration\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"s\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":false}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"cacheTimeout\":null,\"colorBackground\":false,\"colorValue\":false,\"colors\":[\"#299c46\",\"rgba(237, + 129, 40, 0.89)\",\"#d44a3a\"],\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"format\":\"Bps\",\"gauge\":{\"maxValue\":100,\"minValue\":0,\"show\":false,\"thresholdLabels\":false,\"thresholdMarkers\":true},\"gridPos\":{\"h\":4,\"w\":12,\"x\":0,\"y\":8},\"id\":84,\"interval\":null,\"links\":[],\"options\":{\"colorMode\":\"value\",\"graphMode\":\"area\",\"justifyMode\":\"auto\",\"orientation\":\"horizontal\",\"reduceOptions\":{\"calcs\":[\"lastNotNull\"],\"fields\":\"\",\"values\":false},\"textMode\":\"auto\"},\"mappingType\":1,\"mappingTypes\":[{\"name\":\"value + to text\",\"value\":1},{\"name\":\"range to text\",\"value\":2}],\"maxDataPoints\":100,\"nullPointMode\":\"connected\",\"nullText\":null,\"postfix\":\"\",\"postfixFontSize\":\"50%\",\"prefix\":\"\",\"prefixFontSize\":\"50%\",\"rangeMaps\":[{\"from\":\"null\",\"text\":\"N/A\",\"to\":\"null\"}],\"sparkline\":{\"fillColor\":\"rgba(31, + 118, 189, 0.18)\",\"full\":true,\"lineColor\":\"rgb(31, 120, 193)\",\"show\":true},\"tableColumn\":\"\",\"targets\":[{\"expr\":\"sum(irate(istio_tcp_sent_bytes_total{reporter=~\\\"$qrep\\\", + destination_workload_namespace=~\\\"$namespace\\\", destination_workload=~\\\"$workload\\\"}[1m])) + + sum(irate(istio_tcp_received_bytes_total{reporter=~\\\"$qrep\\\", destination_workload_namespace=~\\\"$namespace\\\", + destination_workload=~\\\"$workload\\\"}[1m]))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"\",\"refId\":\"A\"}],\"thresholds\":\"\",\"title\":\"TCP + Server Traffic\",\"type\":\"singlestat\",\"valueFontSize\":\"80%\",\"valueMaps\":[{\"op\":\"=\",\"text\":\"N/A\",\"value\":\"null\"}],\"valueName\":\"avg\"},{\"cacheTimeout\":null,\"colorBackground\":false,\"colorValue\":false,\"colors\":[\"#299c46\",\"rgba(237, + 129, 40, 0.89)\",\"#d44a3a\"],\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"format\":\"Bps\",\"gauge\":{\"maxValue\":100,\"minValue\":0,\"show\":false,\"thresholdLabels\":false,\"thresholdMarkers\":true},\"gridPos\":{\"h\":4,\"w\":12,\"x\":12,\"y\":8},\"id\":85,\"interval\":null,\"links\":[],\"options\":{\"colorMode\":\"value\",\"graphMode\":\"area\",\"justifyMode\":\"auto\",\"orientation\":\"horizontal\",\"reduceOptions\":{\"calcs\":[\"lastNotNull\"],\"fields\":\"\",\"values\":false},\"textMode\":\"auto\"},\"mappingType\":1,\"mappingTypes\":[{\"name\":\"value + to text\",\"value\":1},{\"name\":\"range to text\",\"value\":2}],\"maxDataPoints\":100,\"nullPointMode\":\"connected\",\"nullText\":null,\"postfix\":\"\",\"postfixFontSize\":\"50%\",\"prefix\":\"\",\"prefixFontSize\":\"50%\",\"rangeMaps\":[{\"from\":\"null\",\"text\":\"N/A\",\"to\":\"null\"}],\"sparkline\":{\"fillColor\":\"rgba(31, + 118, 189, 0.18)\",\"full\":true,\"lineColor\":\"rgb(31, 120, 193)\",\"show\":true},\"tableColumn\":\"\",\"targets\":[{\"expr\":\"sum(irate(istio_tcp_sent_bytes_total{reporter=~\\\"$qrep\\\", + source_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$workload\\\"}[1m])) + + sum(irate(istio_tcp_received_bytes_total{reporter=~\\\"$qrep\\\", source_workload_namespace=~\\\"$namespace\\\", + source_workload=~\\\"$workload\\\"}[1m]))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"\",\"refId\":\"A\"}],\"thresholds\":\"\",\"title\":\"TCP + Client Traffic\",\"type\":\"singlestat\",\"valueFontSize\":\"80%\",\"valueMaps\":[{\"op\":\"=\",\"text\":\"N/A\",\"value\":\"null\"}],\"valueName\":\"avg\"}],\"title\":\"General\",\"type\":\"row\"},{\"collapsed\":true,\"gridPos\":{\"h\":1,\"w\":24,\"x\":0,\"y\":1},\"id\":93,\"panels\":[{\"content\":\"
\\nINBOUND WORKLOADS\\n
\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"gridPos\":{\"h\":3,\"w\":24,\"x\":0,\"y\":13},\"id\":45,\"links\":[],\"mode\":\"html\",\"options\":{\"content\":\"
\\nINBOUND WORKLOADS\\n
\",\"mode\":\"html\"},\"pluginVersion\":\"7.1.0\",\"title\":\"\",\"transparent\":true,\"type\":\"text\"},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":0,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":12,\"x\":0,\"y\":16},\"hiddenSeries\":false,\"id\":25,\"legend\":{\"avg\":false,\"current\":false,\"hideEmpty\":true,\"max\":false,\"min\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null + as zero\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"round(sum(irate(istio_requests_total{connection_security_policy=\\\"mutual_tls\\\", + destination_workload_namespace=~\\\"$namespace\\\", destination_workload=~\\\"$workload\\\", + reporter=~\\\"$qrep\\\", source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[5m])) + by (source_workload, source_workload_namespace, response_code), 0.001)\",\"format\":\"time_series\",\"intervalFactor\":1,\"legendFormat\":\"{{ + source_workload }}.{{ source_workload_namespace }} : {{ response_code }} (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"round(sum(irate(istio_requests_total{connection_security_policy!=\\\"mutual_tls\\\", + destination_workload_namespace=~\\\"$namespace\\\", destination_workload=~\\\"$workload\\\", + reporter=~\\\"$qrep\\\", source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[5m])) + by (source_workload, source_workload_namespace, response_code), 0.001)\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + source_workload }}.{{ source_workload_namespace }} : {{ response_code }}\",\"refId\":\"B\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Incoming + Requests By Source And Response Code\",\"tooltip\":{\"shared\":false,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[\"total\"]},\"yaxes\":[{\"format\":\"ops\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":false}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":12,\"x\":12,\"y\":16},\"hiddenSeries\":false,\"id\":26,\"legend\":{\"avg\":false,\"current\":false,\"hideEmpty\":true,\"hideZero\":false,\"max\":false,\"min\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"sum(irate(istio_requests_total{reporter=~\\\"$qrep\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_workload_namespace=~\\\"$namespace\\\", + destination_workload=~\\\"$workload\\\",response_code!~\\\"5.*\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[5m])) by (source_workload, source_workload_namespace) + / sum(irate(istio_requests_total{reporter=~\\\"$qrep\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_workload_namespace=~\\\"$namespace\\\", destination_workload=~\\\"$workload\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[5m])) + by (source_workload, source_workload_namespace)\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + source_workload }}.{{ source_workload_namespace }} (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"sum(irate(istio_requests_total{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_workload_namespace=~\\\"$namespace\\\", + destination_workload=~\\\"$workload\\\",response_code!~\\\"5.*\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[5m])) by (source_workload, source_workload_namespace) + / sum(irate(istio_requests_total{reporter=~\\\"$qrep\\\", connection_security_policy!=\\\"mutual_tls\\\", + destination_workload_namespace=~\\\"$namespace\\\", destination_workload=~\\\"$workload\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[5m])) + by (source_workload, source_workload_namespace)\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + source_workload }}.{{ source_workload_namespace }}\",\"refId\":\"B\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Incoming + Success Rate (non-5xx responses) By Source\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"percentunit\",\"label\":null,\"logBase\":1,\"max\":\"1.01\",\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":false}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"description\":\"\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":8,\"x\":0,\"y\":22},\"hiddenSeries\":false,\"id\":27,\"legend\":{\"alignAsTable\":false,\"avg\":false,\"current\":false,\"hideEmpty\":true,\"hideZero\":false,\"max\":false,\"min\":false,\"rightSide\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"(histogram_quantile(0.50, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[1m])) by (source_workload, source_workload_namespace, + le)) / 1000) or histogram_quantile(0.50, sum(irate(istio_request_duration_seconds_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[1m])) by (source_workload, source_workload_namespace, + le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P50 (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"(histogram_quantile(0.90, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[1m])) by (source_workload, source_workload_namespace, + le)) / 1000) or histogram_quantile(0.90, sum(irate(istio_request_duration_seconds_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[1m])) by (source_workload, source_workload_namespace, + le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P90 (\U0001F510mTLS)\",\"refId\":\"B\",\"step\":2},{\"expr\":\"(histogram_quantile(0.95, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[1m])) by (source_workload, source_workload_namespace, + le)) / 1000) or histogram_quantile(0.95, sum(irate(istio_request_duration_seconds_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[1m])) by (source_workload, source_workload_namespace, + le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P95 (\U0001F510mTLS)\",\"refId\":\"C\",\"step\":2},{\"expr\":\"(histogram_quantile(0.99, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[1m])) by (source_workload, source_workload_namespace, + le)) / 1000) or histogram_quantile(0.99, sum(irate(istio_request_duration_seconds_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[1m])) by (source_workload, source_workload_namespace, + le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P99 (\U0001F510mTLS)\",\"refId\":\"D\",\"step\":2},{\"expr\":\"(histogram_quantile(0.50, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[1m])) by (source_workload, source_workload_namespace, + le)) / 1000) or histogram_quantile(0.50, sum(irate(istio_request_duration_seconds_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[1m])) by (source_workload, source_workload_namespace, + le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P50\",\"refId\":\"E\",\"step\":2},{\"expr\":\"(histogram_quantile(0.90, sum(irate(istio_request_duration_milliseconds_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[1m])) by (source_workload, source_workload_namespace, + le)) / 1000) or histogram_quantile(0.90, sum(irate(istio_request_duration_seconds_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[1m])) by (source_workload, source_workload_namespace, + le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P90\",\"refId\":\"F\",\"step\":2},{\"expr\":\"(histogram_quantile(0.95, sum(irate(istio_request_duration_milliseconds_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[1m])) by (source_workload, source_workload_namespace, + le)) / 1000) or histogram_quantile(0.95, sum(irate(istio_request_duration_seconds_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[1m])) by (source_workload, source_workload_namespace, + le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P95\",\"refId\":\"G\",\"step\":2},{\"expr\":\"(histogram_quantile(0.99, sum(irate(istio_request_duration_milliseconds_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[1m])) by (source_workload, source_workload_namespace, + le)) / 1000) or histogram_quantile(0.99, sum(irate(istio_request_duration_seconds_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[1m])) by (source_workload, source_workload_namespace, + le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P99\",\"refId\":\"H\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Incoming + Request Duration By Source\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"s\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":false}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":8,\"x\":8,\"y\":22},\"hiddenSeries\":false,\"id\":28,\"legend\":{\"alignAsTable\":false,\"avg\":false,\"current\":false,\"hideEmpty\":true,\"max\":false,\"min\":false,\"rightSide\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"histogram_quantile(0.50, + sum(irate(istio_request_bytes_bucket{reporter=~\\\"$qrep\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_workload=~\\\"$workload\\\", destination_workload_namespace=~\\\"$namespace\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P50 (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"histogram_quantile(0.90, + sum(irate(istio_request_bytes_bucket{reporter=~\\\"$qrep\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_workload=~\\\"$workload\\\", destination_workload_namespace=~\\\"$namespace\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + \ P90 (\U0001F510mTLS)\",\"refId\":\"B\",\"step\":2},{\"expr\":\"histogram_quantile(0.95, + sum(irate(istio_request_bytes_bucket{reporter=~\\\"$qrep\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_workload=~\\\"$workload\\\", destination_workload_namespace=~\\\"$namespace\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P95 (\U0001F510mTLS)\",\"refId\":\"C\",\"step\":2},{\"expr\":\"histogram_quantile(0.99, + sum(irate(istio_request_bytes_bucket{reporter=~\\\"$qrep\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_workload=~\\\"$workload\\\", destination_workload_namespace=~\\\"$namespace\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + \ P99 (\U0001F510mTLS)\",\"refId\":\"D\",\"step\":2},{\"expr\":\"histogram_quantile(0.50, + sum(irate(istio_request_bytes_bucket{reporter=~\\\"$qrep\\\", connection_security_policy!=\\\"mutual_tls\\\", + destination_workload=~\\\"$workload\\\", destination_workload_namespace=~\\\"$namespace\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P50\",\"refId\":\"E\",\"step\":2},{\"expr\":\"histogram_quantile(0.90, sum(irate(istio_request_bytes_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[1m])) by (source_workload, source_workload_namespace, + le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P90\",\"refId\":\"F\",\"step\":2},{\"expr\":\"histogram_quantile(0.95, sum(irate(istio_request_bytes_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[1m])) by (source_workload, source_workload_namespace, + le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P95\",\"refId\":\"G\",\"step\":2},{\"expr\":\"histogram_quantile(0.99, sum(irate(istio_request_bytes_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[1m])) by (source_workload, source_workload_namespace, + le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P99\",\"refId\":\"H\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Incoming + Request Size By Source\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"decbytes\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":false}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":8,\"x\":16,\"y\":22},\"hiddenSeries\":false,\"id\":68,\"legend\":{\"alignAsTable\":false,\"avg\":false,\"current\":false,\"hideEmpty\":true,\"max\":false,\"min\":false,\"rightSide\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"histogram_quantile(0.50, + sum(irate(istio_response_bytes_bucket{reporter=~\\\"$qrep\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_workload=~\\\"$workload\\\", destination_workload_namespace=~\\\"$namespace\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P50 (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"histogram_quantile(0.90, + sum(irate(istio_response_bytes_bucket{reporter=~\\\"$qrep\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_workload=~\\\"$workload\\\", destination_workload_namespace=~\\\"$namespace\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + \ P90 (\U0001F510mTLS)\",\"refId\":\"B\",\"step\":2},{\"expr\":\"histogram_quantile(0.95, + sum(irate(istio_response_bytes_bucket{reporter=~\\\"$qrep\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_workload=~\\\"$workload\\\", destination_workload_namespace=~\\\"$namespace\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P95 (\U0001F510mTLS)\",\"refId\":\"C\",\"step\":2},{\"expr\":\"histogram_quantile(0.99, + sum(irate(istio_response_bytes_bucket{reporter=~\\\"$qrep\\\", connection_security_policy=\\\"mutual_tls\\\", + destination_workload=~\\\"$workload\\\", destination_workload_namespace=~\\\"$namespace\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + \ P99 (\U0001F510mTLS)\",\"refId\":\"D\",\"step\":2},{\"expr\":\"histogram_quantile(0.50, + sum(irate(istio_response_bytes_bucket{reporter=~\\\"$qrep\\\", connection_security_policy!=\\\"mutual_tls\\\", + destination_workload=~\\\"$workload\\\", destination_workload_namespace=~\\\"$namespace\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P50\",\"refId\":\"E\",\"step\":2},{\"expr\":\"histogram_quantile(0.90, sum(irate(istio_response_bytes_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[1m])) by (source_workload, source_workload_namespace, + le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P90\",\"refId\":\"F\",\"step\":2},{\"expr\":\"histogram_quantile(0.95, sum(irate(istio_response_bytes_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[1m])) by (source_workload, source_workload_namespace, + le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P95\",\"refId\":\"G\",\"step\":2},{\"expr\":\"histogram_quantile(0.99, sum(irate(istio_response_bytes_bucket{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_workload=~\\\"$workload\\\", + destination_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$srcwl\\\", + source_workload_namespace=~\\\"$srcns\\\"}[1m])) by (source_workload, source_workload_namespace, + le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{source_workload}}.{{source_workload_namespace}} + P99\",\"refId\":\"H\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Response + Size By Source\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"decbytes\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":false}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":12,\"x\":0,\"y\":28},\"hiddenSeries\":false,\"id\":80,\"legend\":{\"avg\":false,\"current\":false,\"max\":false,\"min\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"round(sum(irate(istio_tcp_received_bytes_total{reporter=~\\\"$qrep\\\", + connection_security_policy=\\\"mutual_tls\\\", destination_workload_namespace=~\\\"$namespace\\\", + destination_workload=~\\\"$workload\\\", source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace), 0.001)\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + source_workload }}.{{ source_workload_namespace}} (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"round(sum(irate(istio_tcp_received_bytes_total{reporter=~\\\"$qrep\\\", + connection_security_policy!=\\\"mutual_tls\\\", destination_workload_namespace=~\\\"$namespace\\\", + destination_workload=~\\\"$workload\\\", source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace), 0.001)\",\"format\":\"time_series\",\"intervalFactor\":1,\"legendFormat\":\"{{ + source_workload }}.{{ source_workload_namespace}}\",\"refId\":\"B\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Bytes + Received from Incoming TCP Connection\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"Bps\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":true}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":12,\"x\":12,\"y\":28},\"hiddenSeries\":false,\"id\":82,\"legend\":{\"avg\":false,\"current\":false,\"max\":false,\"min\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"round(sum(irate(istio_tcp_sent_bytes_total{connection_security_policy=\\\"mutual_tls\\\", + reporter=~\\\"$qrep\\\", destination_workload_namespace=~\\\"$namespace\\\", destination_workload=~\\\"$workload\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace), 0.001)\",\"format\":\"time_series\",\"intervalFactor\":1,\"legendFormat\":\"{{ + source_workload }}.{{ source_workload_namespace}} (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"round(sum(irate(istio_tcp_sent_bytes_total{connection_security_policy!=\\\"mutual_tls\\\", + reporter=~\\\"$qrep\\\", destination_workload_namespace=~\\\"$namespace\\\", destination_workload=~\\\"$workload\\\", + source_workload=~\\\"$srcwl\\\", source_workload_namespace=~\\\"$srcns\\\"}[1m])) + by (source_workload, source_workload_namespace), 0.001)\",\"format\":\"time_series\",\"intervalFactor\":1,\"legendFormat\":\"{{ + source_workload }}.{{ source_workload_namespace}}\",\"refId\":\"B\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Bytes + Sent to Incoming TCP Connection\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"Bps\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":true}],\"yaxis\":{\"align\":false,\"alignLevel\":null}}],\"title\":\"Inbound + Workloads\",\"type\":\"row\"},{\"collapsed\":true,\"gridPos\":{\"h\":1,\"w\":24,\"x\":0,\"y\":2},\"id\":91,\"panels\":[{\"content\":\"
\\nOUTBOUND SERVICES\\n
\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"gridPos\":{\"h\":3,\"w\":24,\"x\":0,\"y\":14},\"id\":69,\"links\":[],\"mode\":\"html\",\"options\":{\"content\":\"
\\nOUTBOUND SERVICES\\n
\",\"mode\":\"html\"},\"pluginVersion\":\"7.1.0\",\"title\":\"\",\"transparent\":true,\"type\":\"text\"},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":0,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":12,\"x\":0,\"y\":17},\"hiddenSeries\":false,\"id\":70,\"legend\":{\"avg\":false,\"current\":false,\"hideEmpty\":true,\"max\":false,\"min\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null + as zero\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"round(sum(irate(istio_requests_total{destination_principal=~\\\"spiffe.*\\\", + source_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$workload\\\", + reporter=\\\"source\\\", destination_service=~\\\"$dstsvc\\\"}[5m])) by (destination_service, + response_code), 0.001)\",\"format\":\"time_series\",\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} : {{ response_code }} (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"round(sum(irate(istio_requests_total{destination_principal!~\\\"spiffe.*\\\", + source_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$workload\\\", + reporter=\\\"source\\\", destination_service=~\\\"$dstsvc\\\"}[5m])) by (destination_service, + response_code), 0.001)\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} : {{ response_code }}\",\"refId\":\"B\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Outgoing + Requests By Destination And Response Code\",\"tooltip\":{\"shared\":false,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[\"total\"]},\"yaxes\":[{\"format\":\"ops\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":false}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":12,\"x\":12,\"y\":17},\"hiddenSeries\":false,\"id\":71,\"legend\":{\"avg\":false,\"current\":false,\"hideEmpty\":true,\"hideZero\":false,\"max\":false,\"min\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"sum(irate(istio_requests_total{reporter=\\\"source\\\", + connection_security_policy=\\\"mutual_tls\\\", source_workload_namespace=~\\\"$namespace\\\", + source_workload=~\\\"$workload\\\",response_code!~\\\"5.*\\\", destination_service=~\\\"$dstsvc\\\"}[5m])) + by (destination_service) / sum(irate(istio_requests_total{reporter=\\\"source\\\", + connection_security_policy=\\\"mutual_tls\\\", source_workload_namespace=~\\\"$namespace\\\", + source_workload=~\\\"$workload\\\", destination_service=~\\\"$dstsvc\\\"}[5m])) + by (destination_service)\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"sum(irate(istio_requests_total{reporter=\\\"source\\\", + connection_security_policy!=\\\"mutual_tls\\\", source_workload_namespace=~\\\"$namespace\\\", + source_workload=~\\\"$workload\\\",response_code!~\\\"5.*\\\", destination_service=~\\\"$dstsvc\\\"}[5m])) + by (destination_service) / sum(irate(istio_requests_total{reporter=\\\"source\\\", + connection_security_policy!=\\\"mutual_tls\\\", source_workload_namespace=~\\\"$namespace\\\", + source_workload=~\\\"$workload\\\", destination_service=~\\\"$dstsvc\\\"}[5m])) + by (destination_service)\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }}\",\"refId\":\"B\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Outgoing + Success Rate (non-5xx responses) By Destination\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"percentunit\",\"label\":null,\"logBase\":1,\"max\":\"1.01\",\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":false}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"description\":\"\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":8,\"x\":0,\"y\":23},\"hiddenSeries\":false,\"id\":72,\"legend\":{\"alignAsTable\":false,\"avg\":false,\"current\":false,\"hideEmpty\":true,\"hideZero\":false,\"max\":false,\"min\":false,\"rightSide\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"(histogram_quantile(0.50, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=\\\"source\\\", + connection_security_policy=\\\"mutual_tls\\\", source_workload=~\\\"$workload\\\", + source_workload_namespace=~\\\"$namespace\\\", destination_service=~\\\"$dstsvc\\\"}[1m])) + by (destination_service, le)) / 1000) or histogram_quantile(0.50, sum(irate(istio_request_duration_seconds_bucket{reporter=\\\"source\\\", + connection_security_policy=\\\"mutual_tls\\\", source_workload=~\\\"$workload\\\", + source_workload_namespace=~\\\"$namespace\\\", destination_service=~\\\"$dstsvc\\\"}[1m])) + by (destination_service, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} P50 (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"(histogram_quantile(0.90, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=\\\"source\\\", + connection_security_policy=\\\"mutual_tls\\\", source_workload=~\\\"$workload\\\", + source_workload_namespace=~\\\"$namespace\\\", destination_service=~\\\"$dstsvc\\\"}[1m])) + by (destination_service, le)) / 1000) or histogram_quantile(0.90, sum(irate(istio_request_duration_seconds_bucket{reporter=\\\"source\\\", + connection_security_policy=\\\"mutual_tls\\\", source_workload=~\\\"$workload\\\", + source_workload_namespace=~\\\"$namespace\\\", destination_service=~\\\"$dstsvc\\\"}[1m])) + by (destination_service, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} P90 (\U0001F510mTLS)\",\"refId\":\"B\",\"step\":2},{\"expr\":\"(histogram_quantile(0.95, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=\\\"source\\\", + connection_security_policy=\\\"mutual_tls\\\", source_workload=~\\\"$workload\\\", + source_workload_namespace=~\\\"$namespace\\\", destination_service=~\\\"$dstsvc\\\"}[1m])) + by (destination_service, le)) / 1000) or histogram_quantile(0.95, sum(irate(istio_request_duration_seconds_bucket{reporter=\\\"source\\\", + connection_security_policy=\\\"mutual_tls\\\", source_workload=~\\\"$workload\\\", + source_workload_namespace=~\\\"$namespace\\\", destination_service=~\\\"$dstsvc\\\"}[1m])) + by (destination_service, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} P95 (\U0001F510mTLS)\",\"refId\":\"C\",\"step\":2},{\"expr\":\"(histogram_quantile(0.99, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=\\\"source\\\", + connection_security_policy=\\\"mutual_tls\\\", source_workload=~\\\"$workload\\\", + source_workload_namespace=~\\\"$namespace\\\", destination_service=~\\\"$dstsvc\\\"}[1m])) + by (destination_service, le)) / 1000) or histogram_quantile(0.99, sum(irate(istio_request_duration_seconds_bucket{reporter=\\\"source\\\", + connection_security_policy=\\\"mutual_tls\\\", source_workload=~\\\"$workload\\\", + source_workload_namespace=~\\\"$namespace\\\", destination_service=~\\\"$dstsvc\\\"}[1m])) + by (destination_service, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} P99 (\U0001F510mTLS)\",\"refId\":\"D\",\"step\":2},{\"expr\":\"(histogram_quantile(0.50, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=\\\"source\\\", + connection_security_policy!=\\\"mutual_tls\\\", source_workload=~\\\"$workload\\\", + source_workload_namespace=~\\\"$namespace\\\", destination_service=~\\\"$dstsvc\\\"}[1m])) + by (destination_service, le)) / 1000) or histogram_quantile(0.50, sum(irate(istio_request_duration_seconds_bucket{reporter=\\\"source\\\", + connection_security_policy!=\\\"mutual_tls\\\", source_workload=~\\\"$workload\\\", + source_workload_namespace=~\\\"$namespace\\\", destination_service=~\\\"$dstsvc\\\"}[1m])) + by (destination_service, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} P50\",\"refId\":\"E\",\"step\":2},{\"expr\":\"(histogram_quantile(0.90, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=\\\"source\\\", + connection_security_policy!=\\\"mutual_tls\\\", source_workload=~\\\"$workload\\\", + source_workload_namespace=~\\\"$namespace\\\", destination_service=~\\\"$dstsvc\\\"}[1m])) + by (destination_service, le)) / 1000) or histogram_quantile(0.90, sum(irate(istio_request_duration_seconds_bucket{reporter=\\\"source\\\", + connection_security_policy!=\\\"mutual_tls\\\", source_workload=~\\\"$workload\\\", + source_workload_namespace=~\\\"$namespace\\\", destination_service=~\\\"$dstsvc\\\"}[1m])) + by (destination_service, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} P90\",\"refId\":\"F\",\"step\":2},{\"expr\":\"(histogram_quantile(0.95, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=\\\"source\\\", + connection_security_policy!=\\\"mutual_tls\\\", source_workload=~\\\"$workload\\\", + source_workload_namespace=~\\\"$namespace\\\", destination_service=~\\\"$dstsvc\\\"}[1m])) + by (destination_service, le)) / 1000) or histogram_quantile(0.95, sum(irate(istio_request_duration_seconds_bucket{reporter=\\\"source\\\", + connection_security_policy!=\\\"mutual_tls\\\", source_workload=~\\\"$workload\\\", + source_workload_namespace=~\\\"$namespace\\\", destination_service=~\\\"$dstsvc\\\"}[1m])) + by (destination_service, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} P95\",\"refId\":\"G\",\"step\":2},{\"expr\":\"(histogram_quantile(0.99, + sum(irate(istio_request_duration_milliseconds_bucket{reporter=\\\"source\\\", + connection_security_policy!=\\\"mutual_tls\\\", source_workload=~\\\"$workload\\\", + source_workload_namespace=~\\\"$namespace\\\", destination_service=~\\\"$dstsvc\\\"}[1m])) + by (destination_service, le)) / 1000) or histogram_quantile(0.99, sum(irate(istio_request_duration_seconds_bucket{reporter=\\\"source\\\", + connection_security_policy!=\\\"mutual_tls\\\", source_workload=~\\\"$workload\\\", + source_workload_namespace=~\\\"$namespace\\\", destination_service=~\\\"$dstsvc\\\"}[1m])) + by (destination_service, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} P99\",\"refId\":\"H\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Outgoing + Request Duration By Destination\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"s\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":false}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":8,\"x\":8,\"y\":23},\"hiddenSeries\":false,\"id\":73,\"legend\":{\"alignAsTable\":false,\"avg\":false,\"current\":false,\"hideEmpty\":true,\"max\":false,\"min\":false,\"rightSide\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"histogram_quantile(0.50, + sum(irate(istio_request_bytes_bucket{reporter=\\\"source\\\", connection_security_policy=\\\"mutual_tls\\\", + source_workload=~\\\"$workload\\\", source_workload_namespace=~\\\"$namespace\\\", + destination_service=~\\\"$dstsvc\\\"}[1m])) by (destination_service, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} P50 (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"histogram_quantile(0.90, + sum(irate(istio_request_bytes_bucket{reporter=\\\"source\\\", connection_security_policy=\\\"mutual_tls\\\", + source_workload=~\\\"$workload\\\", source_workload_namespace=~\\\"$namespace\\\", + destination_service=~\\\"$dstsvc\\\"}[1m])) by (destination_service, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} P90 (\U0001F510mTLS)\",\"refId\":\"B\",\"step\":2},{\"expr\":\"histogram_quantile(0.95, + sum(irate(istio_request_bytes_bucket{reporter=\\\"source\\\", connection_security_policy=\\\"mutual_tls\\\", + source_workload=~\\\"$workload\\\", source_workload_namespace=~\\\"$namespace\\\", + destination_service=~\\\"$dstsvc\\\"}[1m])) by (destination_service, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} P95 (\U0001F510mTLS)\",\"refId\":\"C\",\"step\":2},{\"expr\":\"histogram_quantile(0.99, + sum(irate(istio_request_bytes_bucket{reporter=\\\"source\\\", connection_security_policy=\\\"mutual_tls\\\", + source_workload=~\\\"$workload\\\", source_workload_namespace=~\\\"$namespace\\\", + destination_service=~\\\"$dstsvc\\\"}[1m])) by (destination_service, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} P99 (\U0001F510mTLS)\",\"refId\":\"D\",\"step\":2},{\"expr\":\"histogram_quantile(0.50, + sum(irate(istio_request_bytes_bucket{reporter=\\\"source\\\", connection_security_policy!=\\\"mutual_tls\\\", + source_workload=~\\\"$workload\\\", source_workload_namespace=~\\\"$namespace\\\", + destination_service=~\\\"$dstsvc\\\"}[1m])) by (destination_service, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} P50\",\"refId\":\"E\",\"step\":2},{\"expr\":\"histogram_quantile(0.90, + sum(irate(istio_request_bytes_bucket{reporter=\\\"source\\\", connection_security_policy!=\\\"mutual_tls\\\", + source_workload=~\\\"$workload\\\", source_workload_namespace=~\\\"$namespace\\\", + destination_service=~\\\"$dstsvc\\\"}[1m])) by (destination_service, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} P90\",\"refId\":\"F\",\"step\":2},{\"expr\":\"histogram_quantile(0.95, + sum(irate(istio_request_bytes_bucket{reporter=\\\"source\\\", connection_security_policy!=\\\"mutual_tls\\\", + source_workload=~\\\"$workload\\\", source_workload_namespace=~\\\"$namespace\\\", + destination_service=~\\\"$dstsvc\\\"}[1m])) by (destination_service, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} P95\",\"refId\":\"G\",\"step\":2},{\"expr\":\"histogram_quantile(0.99, + sum(irate(istio_request_bytes_bucket{reporter=\\\"source\\\", connection_security_policy!=\\\"mutual_tls\\\", + source_workload=~\\\"$workload\\\", source_workload_namespace=~\\\"$namespace\\\", + destination_service=~\\\"$dstsvc\\\"}[1m])) by (destination_service, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} P99\",\"refId\":\"H\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Outgoing + Request Size By Destination\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"decbytes\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":false}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":8,\"x\":16,\"y\":23},\"hiddenSeries\":false,\"id\":74,\"legend\":{\"alignAsTable\":false,\"avg\":false,\"current\":false,\"hideEmpty\":true,\"max\":false,\"min\":false,\"rightSide\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"histogram_quantile(0.50, + sum(irate(istio_response_bytes_bucket{reporter=\\\"source\\\", connection_security_policy=\\\"mutual_tls\\\", + source_workload=~\\\"$workload\\\", source_workload_namespace=~\\\"$namespace\\\", + destination_service=~\\\"$dstsvc\\\"}[1m])) by (destination_service, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} P50 (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"histogram_quantile(0.90, + sum(irate(istio_response_bytes_bucket{reporter=\\\"source\\\", connection_security_policy=\\\"mutual_tls\\\", + source_workload=~\\\"$workload\\\", source_workload_namespace=~\\\"$namespace\\\", + destination_service=~\\\"$dstsvc\\\"}[1m])) by (destination_service, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} P90 (\U0001F510mTLS)\",\"refId\":\"B\",\"step\":2},{\"expr\":\"histogram_quantile(0.95, + sum(irate(istio_response_bytes_bucket{reporter=\\\"source\\\", connection_security_policy=\\\"mutual_tls\\\", + source_workload=~\\\"$workload\\\", source_workload_namespace=~\\\"$namespace\\\", + destination_service=~\\\"$dstsvc\\\"}[1m])) by (destination_service, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} P95 (\U0001F510mTLS)\",\"refId\":\"C\",\"step\":2},{\"expr\":\"histogram_quantile(0.99, + sum(irate(istio_response_bytes_bucket{reporter=\\\"source\\\", connection_security_policy=\\\"mutual_tls\\\", + source_workload=~\\\"$workload\\\", source_workload_namespace=~\\\"$namespace\\\", + destination_service=~\\\"$dstsvc\\\"}[1m])) by (destination_service, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} P99 (\U0001F510mTLS)\",\"refId\":\"D\",\"step\":2},{\"expr\":\"histogram_quantile(0.50, + sum(irate(istio_response_bytes_bucket{reporter=\\\"source\\\", connection_security_policy!=\\\"mutual_tls\\\", + source_workload=~\\\"$workload\\\", source_workload_namespace=~\\\"$namespace\\\", + destination_service=~\\\"$dstsvc\\\"}[1m])) by (destination_service, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} P50\",\"refId\":\"E\",\"step\":2},{\"expr\":\"histogram_quantile(0.90, + sum(irate(istio_response_bytes_bucket{reporter=\\\"source\\\", connection_security_policy!=\\\"mutual_tls\\\", + source_workload=~\\\"$workload\\\", source_workload_namespace=~\\\"$namespace\\\", + destination_service=~\\\"$dstsvc\\\"}[1m])) by (destination_service, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} P90\",\"refId\":\"F\",\"step\":2},{\"expr\":\"histogram_quantile(0.95, + sum(irate(istio_response_bytes_bucket{reporter=\\\"source\\\", connection_security_policy!=\\\"mutual_tls\\\", + source_workload=~\\\"$workload\\\", source_workload_namespace=~\\\"$namespace\\\", + destination_service=~\\\"$dstsvc\\\"}[1m])) by (destination_service, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} P95\",\"refId\":\"G\",\"step\":2},{\"expr\":\"histogram_quantile(0.99, + sum(irate(istio_response_bytes_bucket{reporter=\\\"source\\\", connection_security_policy!=\\\"mutual_tls\\\", + source_workload=~\\\"$workload\\\", source_workload_namespace=~\\\"$namespace\\\", + destination_service=~\\\"$dstsvc\\\"}[1m])) by (destination_service, le))\",\"format\":\"time_series\",\"hide\":false,\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} P99\",\"refId\":\"H\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Response + Size By Destination\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"decbytes\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":false}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":12,\"x\":0,\"y\":29},\"hiddenSeries\":false,\"id\":76,\"legend\":{\"avg\":false,\"current\":false,\"max\":false,\"min\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"round(sum(irate(istio_tcp_received_bytes_total{connection_security_policy=\\\"mutual_tls\\\", + reporter=\\\"source\\\", source_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$workload\\\", + destination_service=~\\\"$dstsvc\\\"}[1m])) by (destination_service), 0.001)\",\"format\":\"time_series\",\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"round(sum(irate(istio_tcp_received_bytes_total{connection_security_policy!=\\\"mutual_tls\\\", + reporter=\\\"source\\\", source_workload_namespace=~\\\"$namespace\\\", source_workload=~\\\"$workload\\\", + destination_service=~\\\"$dstsvc\\\"}[1m])) by (destination_service), 0.001)\",\"format\":\"time_series\",\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }}\",\"refId\":\"B\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Bytes + Sent on Outgoing TCP Connection\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"Bps\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":true}],\"yaxis\":{\"align\":false,\"alignLevel\":null}},{\"aliasColors\":{},\"bars\":false,\"dashLength\":10,\"dashes\":false,\"datasource\":\"Prometheus\",\"fieldConfig\":{\"defaults\":{\"custom\":{}},\"overrides\":[]},\"fill\":1,\"fillGradient\":0,\"gridPos\":{\"h\":6,\"w\":12,\"x\":12,\"y\":29},\"hiddenSeries\":false,\"id\":78,\"legend\":{\"avg\":false,\"current\":false,\"max\":false,\"min\":false,\"show\":true,\"total\":false,\"values\":false},\"lines\":true,\"linewidth\":1,\"links\":[],\"nullPointMode\":\"null\",\"percentage\":false,\"pluginVersion\":\"7.1.0\",\"pointradius\":5,\"points\":false,\"renderer\":\"flot\",\"seriesOverrides\":[],\"spaceLength\":10,\"stack\":false,\"steppedLine\":false,\"targets\":[{\"expr\":\"round(sum(irate(istio_tcp_sent_bytes_total{reporter=\\\"source\\\", + connection_security_policy=\\\"mutual_tls\\\", source_workload_namespace=~\\\"$namespace\\\", + source_workload=~\\\"$workload\\\", destination_service=~\\\"$dstsvc\\\"}[1m])) + by (destination_service), 0.001)\",\"format\":\"time_series\",\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }} (\U0001F510mTLS)\",\"refId\":\"A\",\"step\":2},{\"expr\":\"round(sum(irate(istio_tcp_sent_bytes_total{reporter=\\\"source\\\", + connection_security_policy!=\\\"mutual_tls\\\", source_workload_namespace=~\\\"$namespace\\\", + source_workload=~\\\"$workload\\\", destination_service=~\\\"$dstsvc\\\"}[1m])) + by (destination_service), 0.001)\",\"format\":\"time_series\",\"intervalFactor\":1,\"legendFormat\":\"{{ + destination_service }}\",\"refId\":\"B\",\"step\":2}],\"thresholds\":[],\"timeFrom\":null,\"timeRegions\":[],\"timeShift\":null,\"title\":\"Bytes + Received from Outgoing TCP Connection\",\"tooltip\":{\"shared\":true,\"sort\":0,\"value_type\":\"individual\"},\"type\":\"graph\",\"xaxis\":{\"buckets\":null,\"mode\":\"time\",\"name\":null,\"show\":true,\"values\":[]},\"yaxes\":[{\"format\":\"Bps\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":\"0\",\"show\":true},{\"format\":\"short\",\"label\":null,\"logBase\":1,\"max\":null,\"min\":null,\"show\":true}],\"yaxis\":{\"align\":false,\"alignLevel\":null}}],\"title\":\"Outbound + Services\",\"type\":\"row\"}],\"refresh\":\"1m\",\"schemaVersion\":26,\"style\":\"dark\",\"tags\":[],\"templating\":{\"list\":[{\"current\":{\"selected\":true,\"text\":\"default\",\"value\":\"default\"},\"hide\":0,\"includeAll\":false,\"label\":null,\"multi\":false,\"name\":\"datasource\",\"options\":[],\"query\":\"prometheus\",\"queryValue\":\"\",\"refresh\":1,\"regex\":\"\",\"skipUrlSync\":false,\"type\":\"datasource\"},{\"allValue\":null,\"current\":{},\"datasource\":\"Prometheus\",\"definition\":\"\",\"hide\":0,\"includeAll\":false,\"label\":\"Namespace\",\"multi\":false,\"name\":\"namespace\",\"options\":[],\"query\":\"query_result(sum(istio_requests_total) + by (destination_workload_namespace) or sum(istio_tcp_sent_bytes_total) by (destination_workload_namespace))\",\"refresh\":1,\"regex\":\"/.*_namespace=\\\"([^\\\"]*).*/\",\"skipUrlSync\":false,\"sort\":0,\"tagValuesQuery\":\"\",\"tags\":[],\"tagsQuery\":\"\",\"type\":\"query\",\"useTags\":false},{\"allValue\":null,\"current\":{},\"datasource\":\"Prometheus\",\"definition\":\"\",\"hide\":0,\"includeAll\":false,\"label\":\"Workload\",\"multi\":false,\"name\":\"workload\",\"options\":[],\"query\":\"query_result((sum(istio_requests_total{destination_workload_namespace=~\\\"$namespace\\\"}) + by (destination_workload) or sum(istio_requests_total{source_workload_namespace=~\\\"$namespace\\\"}) + by (source_workload)) or (sum(istio_tcp_sent_bytes_total{destination_workload_namespace=~\\\"$namespace\\\"}) + by (destination_workload) or sum(istio_tcp_sent_bytes_total{source_workload_namespace=~\\\"$namespace\\\"}) + by (source_workload)))\",\"refresh\":1,\"regex\":\"/.*workload=\\\"([^\\\"]*).*/\",\"skipUrlSync\":false,\"sort\":1,\"tagValuesQuery\":\"\",\"tags\":[],\"tagsQuery\":\"\",\"type\":\"query\",\"useTags\":false},{\"allValue\":null,\"current\":{\"selected\":true,\"text\":\"destination\",\"value\":\"destination\"},\"datasource\":\"Prometheus\",\"definition\":\"\",\"hide\":0,\"includeAll\":false,\"label\":\"Reporter\",\"multi\":true,\"name\":\"qrep\",\"options\":[],\"query\":\"label_values(reporter)\",\"refresh\":1,\"regex\":\"\",\"skipUrlSync\":false,\"sort\":2,\"tagValuesQuery\":\"\",\"tags\":[],\"tagsQuery\":\"\",\"type\":\"query\",\"useTags\":false},{\"allValue\":null,\"current\":{},\"datasource\":\"Prometheus\",\"definition\":\"\",\"hide\":0,\"includeAll\":true,\"label\":\"Inbound + Workload Namespace\",\"multi\":true,\"name\":\"srcns\",\"options\":[],\"query\":\"query_result(sum(istio_requests_total{reporter=~\\\"$qrep\\\", + destination_workload=\\\"$workload\\\", destination_workload_namespace=~\\\"$namespace\\\"}) + by (source_workload_namespace) or sum(istio_tcp_sent_bytes_total{reporter=~\\\"$qrep\\\", + destination_workload=\\\"$workload\\\", destination_workload_namespace=~\\\"$namespace\\\"}) + by (source_workload_namespace))\",\"refresh\":1,\"regex\":\"/.*namespace=\\\"([^\\\"]*).*/\",\"skipUrlSync\":false,\"sort\":2,\"tagValuesQuery\":\"\",\"tags\":[],\"tagsQuery\":\"\",\"type\":\"query\",\"useTags\":false},{\"allValue\":null,\"current\":{},\"datasource\":\"Prometheus\",\"definition\":\"\",\"hide\":0,\"includeAll\":true,\"label\":\"Inbound + Workload\",\"multi\":true,\"name\":\"srcwl\",\"options\":[],\"query\":\"query_result(sum(istio_requests_total{reporter=~\\\"$qrep\\\", + destination_workload=\\\"$workload\\\", destination_workload_namespace=~\\\"$namespace\\\", + source_workload_namespace=~\\\"$srcns\\\"}) by (source_workload) or sum(istio_tcp_sent_bytes_total{reporter=~\\\"$qrep\\\", + destination_workload=\\\"$workload\\\", destination_workload_namespace=~\\\"$namespace\\\", + source_workload_namespace=~\\\"$srcns\\\"}) by (source_workload))\",\"refresh\":1,\"regex\":\"/.*workload=\\\"([^\\\"]*).*/\",\"skipUrlSync\":false,\"sort\":3,\"tagValuesQuery\":\"\",\"tags\":[],\"tagsQuery\":\"\",\"type\":\"query\",\"useTags\":false},{\"allValue\":null,\"current\":{},\"datasource\":\"Prometheus\",\"definition\":\"\",\"hide\":0,\"includeAll\":true,\"label\":\"Destination + Service\",\"multi\":true,\"name\":\"dstsvc\",\"options\":[],\"query\":\"query_result(sum(istio_requests_total{reporter=\\\"source\\\", + source_workload=~\\\"$workload\\\", source_workload_namespace=~\\\"$namespace\\\"}) + by (destination_service) or sum(istio_tcp_sent_bytes_total{reporter=\\\"source\\\", + source_workload=~\\\"$workload\\\", source_workload_namespace=~\\\"$namespace\\\"}) + by (destination_service))\",\"refresh\":1,\"regex\":\"/.*destination_service=\\\"([^\\\"]*).*/\",\"skipUrlSync\":false,\"sort\":4,\"tagValuesQuery\":\"\",\"tags\":[],\"tagsQuery\":\"\",\"type\":\"query\",\"useTags\":false}]},\"time\":{\"from\":\"now-5m\",\"to\":\"now\"},\"timepicker\":{\"refresh_intervals\":[\"5m\",\"15m\",\"30m\",\"1h\",\"2h\",\"1d\"],\"time_options\":[\"5m\",\"15m\",\"1h\",\"6h\",\"12h\",\"24h\",\"2d\",\"7d\",\"30d\"]},\"timezone\":\"\",\"title\":\"Istio + Workload Dashboard\",\"uid\":\"UbsSZTDik\",\"version\":1}\n" +kind: ConfigMap +metadata: + creationTimestamp: null + name: istio-services-grafana-dashboards diff --git a/prometheus/istio/podmonitor-istio-proxy.yaml b/prometheus/istio/podmonitor-istio-proxy.yaml new file mode 100644 index 00000000..1489575a --- /dev/null +++ b/prometheus/istio/podmonitor-istio-proxy.yaml @@ -0,0 +1,13 @@ +apiVersion: monitoring.coreos.com/v1 +kind: PodMonitor +metadata: + name: istio-proxy +spec: + selector: + matchLabels: + security.istio.io/tlsMode: 'istio' + namespaceSelector: + any: true + podMetricsEndpoints: + - port: http-envoy-prom + path: /stats/prometheus diff --git a/prometheus/istio/servicemonitor-istiod.yaml b/prometheus/istio/servicemonitor-istiod.yaml new file mode 100644 index 00000000..caf93cd1 --- /dev/null +++ b/prometheus/istio/servicemonitor-istiod.yaml @@ -0,0 +1,13 @@ +apiVersion: monitoring.coreos.com/v1 +kind: ServiceMonitor +metadata: + name: istiod +spec: + selector: + matchExpressions: + - {key: istio, operator: In, values: [pilot]} + namespaceSelector: + matchNames: + - istio-system + endpoints: + - port: http-monitoring diff --git a/prometheus/values.yaml b/prometheus/values.yaml index 506dfc68..d4e72eb6 100644 --- a/prometheus/values.yaml +++ b/prometheus/values.yaml @@ -3,39 +3,37 @@ prometheusOperator: # exclude the kyma-system Namespace from watching to avoid conflict with the Kyma monitoring stack denyNamespaces: - kyma-system -kubeletService: # re-use the kublet service definition of the Kyma monitoring stack. - enabled: false + kubeletService: + enabled: false + +# change the port of the node-export to be different from the one used by the Kyma monitoring stack prometheus-node-exporter: - # change the port of the node-export to be different from the one used by the Kyma monitoring stack service: port: 9101 targetPort: 9101 ####### This block is disabling control plane components which are not available in the Gardener data plane +# Disable scraping of etcd kubeEtcd: enabled: false + +# Disable scraping of kubeControllerManager kubeControllerManager: enabled: false + +# Disable scraping of kubeProxy kubeProxy: enabled: false + +# Disable scraping of kubeScheduler kubeScheduler: enabled: false -####### This block is needed to also use ServiceMonitors, which are not deployed as part of the chart +####### This block is required to enable scraping of endpoints with Istio strict mTLS, see also https://istio.io/latest/docs/ops/integrations/prometheus/#tls-settings prometheus: prometheusSpec: - # Disable required Helm release labels on ServiceMonitors - serviceMonitorSelectorNilUsesHelmValues: false - # Disable required Helm release labels on PodMonitors - podMonitorSelectorNilUsesHelmValues: false - # Disable required Helm release labels on Probes - probeSelectorNilUsesHelmValues: false - # Disable required Helm release labels on Rules - ruleSelectorNilUsesHelmValues: false - -####### This block is required to enable scraping of endpoints with Istio strict mTLS, see also https://istio.io/latest/docs/ops/integrations/prometheus/#tls-settings podMetadata: annotations: sidecar.istio.io/inject: "true" @@ -46,6 +44,7 @@ prometheus: proxyMetadata: OUTPUT_CERTS: /etc/istio-output-certs sidecar.istio.io/userVolumeMount: '[{"name": "istio-certs", "mountPath": "/etc/istio-output-certs"}]' # mount the shared volume at sidecar proxy + # Additional volumes on the output StatefulSet definition. volumes: - emptyDir: @@ -56,3 +55,113 @@ prometheus: volumeMounts: - mountPath: /etc/prometheus/secrets/istio.default/ name: istio-certs + +####### This block is needed to also use ServiceMonitors, which are not deployed as part of the chart + + # Disable required Helm release labels on ServiceMonitors + serviceMonitorSelectorNilUsesHelmValues: false + # Disable required Helm release labels on PodMonitors + podMonitorSelectorNilUsesHelmValues: false + # Disable required Helm release labels on Probes + probeSelectorNilUsesHelmValues: false + # Disable required Helm release labels on Rules + ruleSelectorNilUsesHelmValues: false + +####### This block enabled scrape discovery based on typical prometheus annotations + + additionalScrapeConfigs: + # Scrape config for service endpoints. + # + # The relabeling allows the actual service scrape endpoint to be configured + # via the following annotations: + # + # * `prometheus.io/scrape`: Only scrape services that have a value of `true` + # * `prometheus.io/scheme`: If the metrics endpoint is secured then you will need + # to set this to `https` & most likely set the `tls_config` of the scrape config. + # * `prometheus.io/path`: If the metrics path is not `/metrics` override this. + # * `prometheus.io/port`: If the metrics are exposed on a different port to the + # service then set this appropriately. + - job_name: 'kubernetes-service-endpoints' + + kubernetes_sd_configs: + - role: endpoints + + relabel_configs: + - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape] + action: keep + regex: true + - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme] + action: replace + target_label: __scheme__ + regex: (https?) + - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path] + action: replace + target_label: __metrics_path__ + regex: (.+) + - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port] + action: replace + target_label: __address__ + regex: ([^:]+)(?::\d+)?;(\d+) + replacement: $1:$2 + - action: labelmap + regex: __meta_kubernetes_service_label_(.+) + - source_labels: [__meta_kubernetes_namespace] + action: replace + target_label: kubernetes_namespace + - source_labels: [__meta_kubernetes_service_name] + action: replace + target_label: kubernetes_name + - source_labels: [__meta_kubernetes_service_name] + action: drop + regex: 'node-exporter' + tls_config: + ca_file: /etc/prometheus/secrets/istio.default/root-cert.pem + cert_file: /etc/prometheus/secrets/istio.default/cert-chain.pem + insecure_skip_verify: true + key_file: /etc/prometheus/secrets/istio.default/key.pem + + # Example scrape config for pods + # + # The relabeling allows the actual pod scrape endpoint to be configured via the + # following annotations: + # + # * `prometheus.io/scrape`: Only scrape pods that have a value of `true` + # * `prometheus.io/scheme`: If the metrics endpoint is secured then you will need + # to set this to `https` & most likely set the `tls_config` of the scrape config. + # * `prometheus.io/path`: If the metrics path is not `/metrics` override this. + # * `prometheus.io/port`: Scrape the pod on the indicated port instead of the default of `9102`. + - job_name: 'kubernetes-pods' + + kubernetes_sd_configs: + - role: pod + + relabel_configs: + - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] + action: keep + regex: true + - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme] + action: replace + target_label: __scheme__ + regex: (https?) + - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] + action: replace + target_label: __metrics_path__ + regex: (.+) + - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port] + action: replace + regex: ([^:]+)(?::\d+)?;(\d+) + replacement: $1:$2 + target_label: __address__ + - action: labelmap + regex: __meta_kubernetes_pod_label_(.+) + - source_labels: [__meta_kubernetes_namespace] + action: replace + target_label: kubernetes_namespace + - source_labels: [__meta_kubernetes_pod_name] + action: replace + target_label: kubernetes_pod_name + tls_config: + ca_file: /etc/prometheus/secrets/istio.default/root-cert.pem + cert_file: /etc/prometheus/secrets/istio.default/cert-chain.pem + insecure_skip_verify: true + key_file: /etc/prometheus/secrets/istio.default/key.pem From 2d7d1bf58aa97ac05908c33ef65c4a9fb59fa749 Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Fri, 23 Sep 2022 17:41:26 +0200 Subject: [PATCH 09/30] fix --- .../istio/configmap-istio-grafana-dashboards.yaml | 10 ++++++---- .../configmap-istio-services-grafana-dashboards.yaml | 10 ++++++---- prometheus/values.yaml | 6 ++++++ 3 files changed, 18 insertions(+), 8 deletions(-) diff --git a/prometheus/istio/configmap-istio-grafana-dashboards.yaml b/prometheus/istio/configmap-istio-grafana-dashboards.yaml index dda91e3e..0ff00a43 100644 --- a/prometheus/istio/configmap-istio-grafana-dashboards.yaml +++ b/prometheus/istio/configmap-istio-grafana-dashboards.yaml @@ -1,10 +1,12 @@ apiVersion: v1 +kind: ConfigMap +metadata: + labels: + grafana_dashboard: "1" + name: istio-grafana-dashboards data: istio-performance-dashboard.json: | {"annotations":{"list":[{"builtIn":1,"datasource":"-- Grafana --","enable":true,"hide":true,"iconColor":"rgba(0, 211, 255, 1)","name":"Annotations & Alerts","type":"dashboard"}]},"editable":false,"gnetId":null,"graphTooltip":0,"links":[],"panels":[{"collapsed":true,"gridPos":{"h":1,"w":24,"x":0,"y":0},"id":21,"panels":[{"content":"The charts on this dashboard are intended to show Istio main components cost in terms of resources utilization under steady load.\n\n- **vCPU / 1k rps:** shows vCPU utilization by the main Istio components normalized by 1000 requests/second. When idle or low traffic, this chart will be blank. The curve for istio-proxy refers to the services sidecars only.\n- **vCPU:** vCPU utilization by Istio components, not normalized.\n- **Memory:** memory footprint for the components. Telemetry and policy are normalized by 1k rps, and no data is shown when there is no traffic. For ingress and istio-proxy, the data is per instance.\n- **Bytes transferred / sec:** shows the number of bytes flowing through each Istio component.\n\n\n","gridPos":{"h":6,"w":24,"x":0,"y":1},"id":19,"links":[],"mode":"markdown","timeFrom":null,"timeShift":null,"title":"Performance Dashboard README","transparent":true,"type":"text"}],"title":"Performance Dashboard Notes","type":"row"},{"collapsed":false,"gridPos":{"h":1,"w":24,"x":0,"y":1},"id":6,"panels":[],"title":"vCPU Usage","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":8,"w":12,"x":0,"y":2},"id":4,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"(sum(irate(container_cpu_usage_seconds_total{pod=~\"istio-ingressgateway-.*\",container=\"istio-proxy\"}[1m])) / (round(sum(irate(istio_requests_total{source_workload=\"istio-ingressgateway\", reporter=\"source\"}[1m])), 0.001)/1000))","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"istio-ingressgateway","refId":"A"},{"expr":"(sum(irate(container_cpu_usage_seconds_total{namespace!=\"istio-system\",container=\"istio-proxy\"}[1m]))/ (round(sum(irate(istio_requests_total[1m])), 0.001)/1000))/ (sum(irate(istio_requests_total{source_workload=\"istio-ingressgateway\"}[1m])) >bool 10)","format":"time_series","intervalFactor":1,"legendFormat":"istio-proxy","refId":"B"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"vCPU / 1k rps","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":8,"w":12,"x":12,"y":2},"id":7,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(rate(container_cpu_usage_seconds_total{pod=~\"istio-ingressgateway-.*\",container=\"istio-proxy\"}[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"istio-ingressgateway","refId":"A"},{"expr":"sum(rate(container_cpu_usage_seconds_total{namespace!=\"istio-system\",container=\"istio-proxy\"}[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"istio-proxy","refId":"B"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"vCPU","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"gridPos":{"h":1,"w":24,"x":0,"y":10},"id":13,"panels":[],"title":"Memory and Data Rates","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":8,"w":12,"x":0,"y":11},"id":902,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(container_memory_working_set_bytes{pod=~\"istio-ingressgateway-.*\"}) / count(container_memory_working_set_bytes{pod=~\"istio-ingressgateway-.*\",container!=\"POD\"})","format":"time_series","intervalFactor":1,"legendFormat":"per istio-ingressgateway","refId":"A"},{"expr":"sum(container_memory_working_set_bytes{namespace!=\"istio-system\",container=\"istio-proxy\"}) / count(container_memory_working_set_bytes{namespace!=\"istio-system\",container=\"istio-proxy\"})","format":"time_series","intervalFactor":1,"legendFormat":"per istio proxy","refId":"B"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Memory Usage","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bytes","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":8,"w":12,"x":12,"y":11},"id":11,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(irate(istio_response_bytes_sum{source_workload=\"istio-ingressgateway\", reporter=\"source\"}[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"istio-ingressgateway","refId":"A"},{"expr":"sum(irate(istio_response_bytes_sum{source_workload_namespace!=\"istio-system\", reporter=\"source\"}[1m])) + sum(irate(istio_request_bytes_sum{source_workload_namespace!=\"istio-system\", reporter=\"source\"}[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"istio-proxy","refId":"B"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Bytes transferred / sec","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"Bps","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"gridPos":{"h":1,"w":24,"x":0,"y":19},"id":17,"panels":[],"title":"Istio Component Versions","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":8,"w":24,"x":0,"y":20},"id":15,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(istio_build) by (component, tag)","format":"time_series","intervalFactor":1,"legendFormat":"{{ component }}: {{ tag }}","refId":"A"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Istio Components by Version","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"gridPos":{"h":1,"w":24,"x":0,"y":31},"id":71,"panels":[],"title":"Proxy Resource Usage","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":7,"w":6,"x":0,"y":32},"id":72,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(container_memory_working_set_bytes{container=\"istio-proxy\"})","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"Total (k8s)","refId":"A","step":2}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Memory","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bytes","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":7,"w":6,"x":6,"y":32},"id":73,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(rate(container_cpu_usage_seconds_total{container=\"istio-proxy\"}[1m]))","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"Total (k8s)","refId":"A","step":2}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"vCPU","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":7,"w":6,"x":12,"y":32},"id":702,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(container_fs_usage_bytes{container=\"istio-proxy\"})","format":"time_series","intervalFactor":2,"legendFormat":"Total (k8s)","refId":"A","step":2}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Disk","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bytes","label":"","logBase":1,"max":null,"min":null,"show":true},{"decimals":null,"format":"none","label":"","logBase":1024,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"gridPos":{"h":1,"w":24,"x":0,"y":39},"id":69,"panels":[],"title":"Istiod Resource Usage","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":7,"w":6,"x":0,"y":40},"id":5,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"process_virtual_memory_bytes{app=\"istiod\"}","format":"time_series","instant":false,"intervalFactor":2,"legendFormat":"Virtual Memory","refId":"I","step":2},{"expr":"process_resident_memory_bytes{app=\"istiod\"}","format":"time_series","intervalFactor":2,"legendFormat":"Resident Memory","refId":"H","step":2},{"expr":"go_memstats_heap_sys_bytes{app=\"istiod\"}","format":"time_series","hide":true,"intervalFactor":2,"legendFormat":"heap sys","refId":"A"},{"expr":"go_memstats_heap_alloc_bytes{app=\"istiod\"}","format":"time_series","hide":true,"intervalFactor":2,"legendFormat":"heap alloc","refId":"D"},{"expr":"go_memstats_alloc_bytes{app=\"istiod\"}","format":"time_series","intervalFactor":2,"legendFormat":"Alloc","refId":"F","step":2},{"expr":"go_memstats_heap_inuse_bytes{app=\"istiod\"}","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"Heap in-use","refId":"E","step":2},{"expr":"go_memstats_stack_inuse_bytes{app=\"istiod\"}","format":"time_series","intervalFactor":2,"legendFormat":"Stack in-use","refId":"G","step":2},{"expr":"sum(container_memory_working_set_bytes{container=~\"discovery|istio-proxy\", pod=~\"istiod-.*\"})","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"Total (k8s)","refId":"C","step":2},{"expr":"container_memory_working_set_bytes{container=~\"discovery|istio-proxy\", pod=~\"istiod-.*\"}","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"{{ container }} (k8s)","refId":"B","step":2}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Memory","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bytes","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":7,"w":6,"x":6,"y":40},"id":602,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(rate(container_cpu_usage_seconds_total{container=~\"discovery|istio-proxy\", pod=~\"istiod-.*\"}[1m]))","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"Total (k8s)","refId":"A","step":2},{"expr":"sum(rate(container_cpu_usage_seconds_total{container=~\"discovery|istio-proxy\", pod=~\"istiod-.*\"}[1m])) by (container)","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"{{ container }} (k8s)","refId":"B","step":2},{"expr":"irate(process_cpu_seconds_total{app=\"istiod\"}[1m])","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"pilot (self-reported)","refId":"C","step":2}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"vCPU","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":7,"w":6,"x":12,"y":40},"id":74,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"process_open_fds{app=\"istiod\"}","format":"time_series","hide":true,"instant":false,"interval":"","intervalFactor":2,"legendFormat":"Open FDs (pilot)","refId":"A"},{"expr":"container_fs_usage_bytes{ container=~\"discovery|istio-proxy\", pod=~\"istiod-.*\"}","format":"time_series","intervalFactor":2,"legendFormat":"{{ container }}","refId":"B","step":2}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Disk","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bytes","label":"","logBase":1,"max":null,"min":null,"show":true},{"decimals":null,"format":"none","label":"","logBase":1024,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":7,"w":6,"x":18,"y":40},"id":402,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":false,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"go_goroutines{app=\"istiod\"}","format":"time_series","intervalFactor":2,"legendFormat":"Number of Goroutines","refId":"A","step":2}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Goroutines","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":"","logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}}],"refresh":"10s","schemaVersion":18,"style":"dark","tags":[],"templating":{"list":[]},"time":{"from":"now-5m","to":"now"},"timepicker":{"refresh_intervals":["5s","10s","30s","1m","5m","15m","30m","1h","2h","1d"],"time_options":["5m","15m","1h","6h","12h","24h","2d","7d","30d"]},"timezone":"","title":"Istio Performance Dashboard","uid":"vu8e0VWZk","version":22} pilot-dashboard.json: | {"annotations":{"list":[{"builtIn":1,"datasource":"-- Grafana --","enable":true,"hide":true,"iconColor":"rgba(0, 211, 255, 1)","name":"Annotations & Alerts","type":"dashboard"}]},"editable":false,"gnetId":null,"graphTooltip":1,"links":[],"panels":[{"collapsed":false,"gridPos":{"h":1,"w":24,"x":0,"y":0},"id":60,"panels":[],"title":"Deployed Versions","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":5,"w":24,"x":0,"y":1},"id":56,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(istio_build{component=\"pilot\"}) by (tag)","format":"time_series","intervalFactor":1,"legendFormat":"{{ tag }}","refId":"A"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Pilot Versions","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"gridPos":{"h":1,"w":24,"x":0,"y":6},"id":62,"panels":[],"title":"Resource Usage","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":7,"w":6,"x":0,"y":7},"id":5,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"process_virtual_memory_bytes{app=\"istiod\"}","format":"time_series","instant":false,"intervalFactor":2,"legendFormat":"Virtual Memory","refId":"I","step":2},{"expr":"process_resident_memory_bytes{app=\"istiod\"}","format":"time_series","intervalFactor":2,"legendFormat":"Resident Memory","refId":"H","step":2},{"expr":"go_memstats_heap_sys_bytes{app=\"istiod\"}","format":"time_series","hide":true,"intervalFactor":2,"legendFormat":"heap sys","refId":"A"},{"expr":"go_memstats_heap_alloc_bytes{app=\"istiod\"}","format":"time_series","hide":true,"intervalFactor":2,"legendFormat":"heap alloc","refId":"D"},{"expr":"go_memstats_alloc_bytes{app=\"istiod\"}","format":"time_series","intervalFactor":2,"legendFormat":"Alloc","refId":"F","step":2},{"expr":"go_memstats_heap_inuse_bytes{app=\"istiod\"}","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"Heap in-use","refId":"E","step":2},{"expr":"go_memstats_stack_inuse_bytes{app=\"istiod\"}","format":"time_series","intervalFactor":2,"legendFormat":"Stack in-use","refId":"G","step":2},{"expr":"container_memory_working_set_bytes{container=~\"discovery\", pod=~\"istiod-.*|istio-pilot-.*\"}","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"Discovery (container)","refId":"B","step":2},{"expr":"container_memory_working_set_bytes{container=~\"istio-proxy\", pod=~\"istiod-.*|istio-pilot-.*\"}","format":"time_series","intervalFactor":1,"legendFormat":"Sidecar (container)","refId":"C"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Memory","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bytes","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":7,"w":6,"x":6,"y":7},"id":6,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(irate(container_cpu_usage_seconds_total{container=\"discovery\", pod=~\"istiod-.*|istio-pilot-.*\"}[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"Discovery (container)","refId":"A"},{"expr":"irate(process_cpu_seconds_total{app=\"istiod\"}[1m])","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"Discovery (process)","refId":"C","step":2},{"expr":"sum(irate(container_cpu_usage_seconds_total{container=\"istio-proxy\", pod=~\"istiod-.*|istio-pilot-.*\"}[1m]))","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"Sidecar (container)","refId":"B","step":2}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"CPU","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":7,"w":6,"x":12,"y":7},"id":7,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"container_fs_usage_bytes{container=\"discovery\", pod=~\"istiod-.*|istio-pilot-.*\"}","format":"time_series","intervalFactor":2,"legendFormat":"Discovery","refId":"B","step":2},{"expr":"container_fs_usage_bytes{container=\"istio-proxy\", pod=~\"istiod-.*|istio-pilot-.*\"}","format":"time_series","intervalFactor":1,"legendFormat":"Sidecar","refId":"A"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Disk","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"bytes","label":"","logBase":1,"max":null,"min":null,"show":true},{"decimals":null,"format":"none","label":"","logBase":1024,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":7,"w":6,"x":18,"y":7},"id":4,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":false,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"go_goroutines{app=\"istiod\"}","format":"time_series","intervalFactor":2,"legendFormat":"Number of Goroutines","refId":"A","step":2}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Goroutines","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":"","logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"gridPos":{"h":1,"w":24,"x":0,"y":14},"id":58,"panels":[],"title":"Pilot Push Information","type":"row"},{"aliasColors":{},"bars":true,"dashLength":10,"dashes":false,"datasource":"Prometheus","description":"Shows the rate of pilot pushes","fill":1,"gridPos":{"h":8,"w":8,"x":0,"y":15},"id":622,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":false,"linewidth":1,"links":[],"nullPointMode":"null as zero","paceLength":10,"percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":true,"steppedLine":false,"targets":[{"expr":"sum(irate(pilot_xds_pushes{type=\"cds\"}[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"Cluster","refId":"C"},{"expr":"sum(irate(pilot_xds_pushes{type=\"eds\"}[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"Endpoints","refId":"D"},{"expr":"sum(irate(pilot_xds_pushes{type=\"lds\"}[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"Listeners","refId":"A"},{"expr":"sum(irate(pilot_xds_pushes{type=\"rds\"}[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"Routes","refId":"E"},{"expr":"sum(irate(pilot_xds_pushes{type=\"sds\"}[1m]))","interval":"","legendFormat":"Secrets","refId":"B"},{"expr":"sum(irate(pilot_xds_pushes{type=\"nds\"}[1m]))","interval":"","legendFormat":"Nametables","refId":"F"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Pilot Pushes","tooltip":{"shared":false,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":["total"]},"yaxes":[{"format":"ops","label":null,"logBase":1,"max":null,"min":"0","show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","description":"Captures a variety of pilot errors","fill":1,"gridPos":{"h":8,"w":8,"x":8,"y":15},"id":67,"legend":{"avg":false,"current":false,"hideEmpty":true,"hideZero":true,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(pilot_xds_cds_reject{app=\"istiod\"}) or (absent(pilot_xds_cds_reject{app=\"istiod\"}) - 1)","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"Rejected CDS Configs","refId":"C"},{"expr":"sum(pilot_xds_eds_reject{app=\"istiod\"}) or (absent(pilot_xds_eds_reject{app=\"istiod\"}) - 1)","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"Rejected EDS Configs","refId":"D"},{"expr":"sum(pilot_xds_rds_reject{app=\"istiod\"}) or (absent(pilot_xds_rds_reject{app=\"istiod\"}) - 1)","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"Rejected RDS Configs","refId":"A"},{"expr":"sum(pilot_xds_lds_reject{app=\"istiod\"}) or (absent(pilot_xds_lds_reject{app=\"istiod\"}) - 1)","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"Rejected LDS Configs","refId":"B"},{"expr":"sum(rate(pilot_xds_write_timeout{app=\"istiod\"}[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"Write Timeouts","refId":"F"},{"expr":"sum(rate(pilot_total_xds_internal_errors{app=\"istiod\"}[1m]))","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"Internal Errors","refId":"H"},{"expr":"sum(rate(pilot_total_xds_rejects{app=\"istiod\"}[1m]))","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"Config Rejection Rate","refId":"E"},{"expr":"sum(rate(pilot_xds_push_context_errors{app=\"istiod\"}[1m]))","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"Push Context Errors","refId":"K"},{"expr":"sum(rate(pilot_xds_write_timeout{app=\"istiod\"}[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"Push Timeouts","refId":"G"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Pilot Errors","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","description":"Shows the total time it takes to push a config update to a proxy","fill":1,"gridPos":{"h":8,"w":8,"x":16,"y":15},"id":624,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"histogram_quantile(0.5, sum(rate(pilot_proxy_convergence_time_bucket[1m])) by (le))","format":"time_series","intervalFactor":1,"legendFormat":"p50 ","refId":"A"},{"expr":"histogram_quantile(0.9, sum(rate(pilot_proxy_convergence_time_bucket[1m])) by (le))","format":"time_series","intervalFactor":1,"legendFormat":"p90","refId":"B"},{"expr":"histogram_quantile(0.99, sum(rate(pilot_proxy_convergence_time_bucket[1m])) by (le))","format":"time_series","intervalFactor":1,"legendFormat":"p99","refId":"C"},{"expr":"histogram_quantile(0.999, sum(rate(pilot_proxy_convergence_time_bucket[1m])) by (le))","format":"time_series","intervalFactor":1,"legendFormat":"p99.9","refId":"D"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Proxy Push Time","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"s","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":8,"w":12,"x":0,"y":23},"id":45,"legend":{"avg":false,"current":false,"hideEmpty":true,"hideZero":true,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null as zero","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"pilot_conflict_inbound_listener{app=\"istiod\"}","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"Inbound Listeners","refId":"B"},{"expr":"pilot_conflict_outbound_listener_http_over_current_tcp{app=\"istiod\"}","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"Outbound Listeners (http over current tcp)","refId":"A"},{"expr":"pilot_conflict_outbound_listener_tcp_over_current_tcp{app=\"istiod\"}","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"Outbound Listeners (tcp over current tcp)","refId":"C"},{"expr":"pilot_conflict_outbound_listener_tcp_over_current_http{app=\"istiod\"}","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"Outbound Listeners (tcp over current http)","refId":"D"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Conflicts","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":8,"w":12,"x":12,"y":23},"id":47,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"avg(pilot_virt_services{app=\"istiod\"})","format":"time_series","intervalFactor":1,"legendFormat":"Virtual Services","refId":"A"},{"expr":"avg(pilot_services{app=\"istiod\"})","format":"time_series","intervalFactor":1,"legendFormat":"Services","refId":"B"},{"expr":"sum(pilot_xds{app=\"istiod\"}) by (pod)","format":"time_series","intervalFactor":1,"legendFormat":"Connected Endpoints {{pod}}","refId":"E"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"ADS Monitoring","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"gridPos":{"h":1,"w":24,"x":0,"y":31},"id":64,"panels":[],"title":"Envoy Information","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","description":"Shows details about Envoy proxies in the mesh","fill":1,"gridPos":{"h":8,"w":8,"x":0,"y":32},"id":40,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(irate(envoy_cluster_upstream_cx_total{cluster_name=\"xds-grpc\"}[1m]))","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"XDS Connections","refId":"C"},{"expr":"sum(irate(envoy_cluster_upstream_cx_connect_fail{cluster_name=\"xds-grpc\"}[1m]))","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"XDS Connection Failures","refId":"A"},{"expr":"sum(increase(envoy_server_hot_restart_epoch[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"Envoy Restarts","refId":"B"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Envoy Details","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"ops","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"ops","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fill":1,"gridPos":{"h":8,"w":8,"x":8,"y":32},"id":41,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(envoy_cluster_upstream_cx_active{cluster_name=\"xds-grpc\"})","format":"time_series","intervalFactor":2,"legendFormat":"XDS Active Connections","refId":"C","step":2}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"XDS Active Connections","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","description":"Shows the size of XDS requests and responses","fill":1,"gridPos":{"h":8,"w":8,"x":16,"y":32},"id":42,"legend":{"avg":false,"current":false,"hideEmpty":false,"hideZero":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","percentage":false,"pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"max(rate(envoy_cluster_upstream_cx_rx_bytes_total{cluster_name=\"xds-grpc\"}[1m]))","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"XDS Response Bytes Max","refId":"D"},{"expr":"quantile(0.5, rate(envoy_cluster_upstream_cx_rx_bytes_total{cluster_name=\"xds-grpc\"}[1m]))","format":"time_series","hide":false,"intervalFactor":1,"legendFormat":"XDS Response Bytes Average","refId":"B"},{"expr":"max(rate(envoy_cluster_upstream_cx_tx_bytes_total{cluster_name=\"xds-grpc\"}[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"XDS Request Bytes Max","refId":"A"},{"expr":"quantile(.5, rate(envoy_cluster_upstream_cx_tx_bytes_total{cluster_name=\"xds-grpc\"}[1m]))","format":"time_series","intervalFactor":1,"legendFormat":"XDS Request Bytes Average","refId":"C"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"XDS Requests Size","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"Bps","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"ops","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"datasource":null,"gridPos":{"h":1,"w":24,"x":0,"y":40},"id":626,"panels":[],"title":"Webhooks","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":null,"fill":1,"fillGradient":0,"gridPos":{"h":8,"w":12,"x":0,"y":41},"hiddenSeries":false,"id":629,"legend":{"avg":false,"current":false,"hideEmpty":false,"hideZero":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"nullPointMode":"null","options":{"dataLinks":[]},"percentage":false,"pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(rate(galley_validation_passed[1m]))","interval":"","legendFormat":"Validations (Success)","refId":"A"},{"expr":"sum(rate(galley_validation_failed[1m]))","interval":"","legendFormat":"Validation (Failure)","refId":"B"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Configuration Validation","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":null,"description":"","fill":1,"fillGradient":0,"gridPos":{"h":8,"w":12,"x":12,"y":41},"hiddenSeries":false,"id":630,"legend":{"avg":false,"current":false,"hideZero":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"nullPointMode":"null","options":{"dataLinks":[]},"percentage":false,"pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(rate(sidecar_injection_success_total[1m]))","interval":"","legendFormat":"Injections (Success)","refId":"A"},{"expr":"sum(rate(sidecar_injection_failure_total[1m]))","interval":"","legendFormat":"Injections (Failure)","refId":"B"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Sidecar Injection","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}}],"refresh":"5s","schemaVersion":18,"style":"dark","tags":[],"templating":{"list":[{"current":{"selected":true,"text":"default","value":"default"},"hide":0,"includeAll":false,"label":null,"multi":false,"name":"datasource","options":[],"query":"prometheus","queryValue":"","refresh":1,"regex":"","skipUrlSync":false,"type":"datasource"}]},"time":{"from":"now-5m","to":"now"},"timepicker":{"refresh_intervals":["5s","10s","30s","1m","5m","15m","30m","1h","2h","1d"],"time_options":["5m","15m","1h","6h","12h","24h","2d","7d","30d"]},"timezone":"browser","title":"Istio Control Plane Dashboard","uid":"3--MLVZZk","version":11} -kind: ConfigMap -metadata: - creationTimestamp: null - name: istio-grafana-dashboards + diff --git a/prometheus/istio/configmap-istio-services-grafana-dashboards.yaml b/prometheus/istio/configmap-istio-services-grafana-dashboards.yaml index 369c36ea..a57e32d9 100644 --- a/prometheus/istio/configmap-istio-services-grafana-dashboards.yaml +++ b/prometheus/istio/configmap-istio-services-grafana-dashboards.yaml @@ -1,4 +1,9 @@ apiVersion: v1 +kind: ConfigMap +metadata: + labels: + grafana_dashboard: "1" + name: istio-services-grafana-dashboards data: istio-extension-dashboard.json: | {"annotations":{"list":[{"builtIn":1,"datasource":"-- Grafana --","enable":true,"hide":true,"iconColor":"rgba(0, 211, 255, 1)","name":"Annotations & Alerts","type":"dashboard"}]},"editable":false,"gnetId":null,"graphTooltip":0,"links":[],"panels":[{"collapsed":false,"datasource":"Prometheus","gridPos":{"h":1,"w":24,"x":0,"y":0},"id":3,"panels":[],"title":"Wasm VMs","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","description":"","fieldConfig":{"defaults":{"custom":{"align":null},"links":[],"mappings":[],"thresholds":{"mode":"absolute","steps":[{"color":"green","value":null},{"color":"red","value":80}]}},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":8,"w":12,"x":0,"y":1},"hiddenSeries":false,"id":2,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"nullPointMode":"null","options":{"alertThreshold":true},"percentage":false,"pluginVersion":"7.2.1","pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"avg(envoy_wasm_envoy_wasm_runtime_null_active)","interval":"","legendFormat":"native","refId":"A"},{"expr":"avg(envoy_wasm_envoy_wasm_runtime_v8_active)","interval":"","legendFormat":"v8","refId":"B"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Active","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"$$hashKey":"object:123","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"$$hashKey":"object:124","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fieldConfig":{"defaults":{"custom":{},"links":[]},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":8,"w":12,"x":12,"y":1},"hiddenSeries":false,"id":6,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"nullPointMode":"null","options":{"alertThreshold":true},"percentage":false,"pluginVersion":"7.2.1","pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"avg(envoy_wasm_envoy_wasm_runtime_null_created)","interval":"","legendFormat":"native","refId":"A"},{"expr":"avg(envoy_wasm_envoy_wasm_runtime_v8_created)","interval":"","legendFormat":"v8","refId":"B"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Created","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"$$hashKey":"object:68","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"$$hashKey":"object:69","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"datasource":"Prometheus","gridPos":{"h":1,"w":24,"x":0,"y":9},"id":7,"panels":[],"title":"Wasm Module Remote Load","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fieldConfig":{"defaults":{"custom":{},"links":[]},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":8,"w":8,"x":0,"y":10},"hiddenSeries":false,"id":11,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"nullPointMode":"null","options":{"alertThreshold":true},"percentage":false,"pluginVersion":"7.2.1","pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"avg(envoy_wasm_remote_load_cache_entries)","interval":"","legendFormat":"entries","refId":"A"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Cache Entry","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"$$hashKey":"object:178","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"$$hashKey":"object:179","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fieldConfig":{"defaults":{"custom":{},"links":[]},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":8,"w":8,"x":8,"y":10},"hiddenSeries":false,"id":8,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"nullPointMode":"null","options":{"alertThreshold":true},"percentage":false,"pluginVersion":"7.2.1","pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"avg(envoy_wasm_remote_load_cache_hits)","interval":"","legendFormat":"hits","refId":"A"},{"expr":"avg(envoy_wasm_remote_load_cache_misses)","interval":"","legendFormat":"misses","refId":"B"},{"expr":"avg(envoy_wasm_remote_load_cache_negative_hits)","interval":"","legendFormat":"negative hits","refId":"C"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Cache Visit","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"$$hashKey":"object:233","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"$$hashKey":"object:234","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fieldConfig":{"defaults":{"custom":{},"links":[]},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":8,"w":8,"x":16,"y":10},"hiddenSeries":false,"id":10,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"nullPointMode":"null","options":{"alertThreshold":true},"percentage":false,"pluginVersion":"7.2.1","pointradius":2,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"avg(envoy_wasm_remote_load_fetch_failures)","interval":"","legendFormat":"failures","refId":"A"},{"expr":"avg(envoy_wasm_remote_load_fetch_successes)","interval":"","legendFormat":"successes","refId":"B"}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Remote Fetch","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"$$hashKey":"object:288","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"$$hashKey":"object:289","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}},{"collapsed":false,"datasource":"Prometheus","gridPos":{"h":1,"w":24,"x":0,"y":18},"id":71,"panels":[],"title":"Proxy Resource Usage","type":"row"},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":8,"w":12,"x":0,"y":19},"hiddenSeries":false,"id":72,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","options":{"alertThreshold":true},"percentage":false,"pluginVersion":"7.2.1","pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(container_memory_working_set_bytes{container=\"istio-proxy\"})","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"Total (k8s)","refId":"A","step":2}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"Memory","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"$$hashKey":"object:396","format":"bytes","label":null,"logBase":1,"max":null,"min":null,"show":true},{"$$hashKey":"object:397","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":false}],"yaxis":{"align":false,"alignLevel":null}},{"aliasColors":{},"bars":false,"dashLength":10,"dashes":false,"datasource":"Prometheus","fieldConfig":{"defaults":{"custom":{}},"overrides":[]},"fill":1,"fillGradient":0,"gridPos":{"h":8,"w":12,"x":12,"y":19},"hiddenSeries":false,"id":73,"legend":{"avg":false,"current":false,"max":false,"min":false,"show":true,"total":false,"values":false},"lines":true,"linewidth":1,"links":[],"nullPointMode":"null","options":{"alertThreshold":true},"percentage":false,"pluginVersion":"7.2.1","pointradius":5,"points":false,"renderer":"flot","seriesOverrides":[],"spaceLength":10,"stack":false,"steppedLine":false,"targets":[{"expr":"sum(rate(container_cpu_usage_seconds_total{container=\"istio-proxy\"}[1m]))","format":"time_series","hide":false,"intervalFactor":2,"legendFormat":"Total (k8s)","refId":"A","step":2}],"thresholds":[],"timeFrom":null,"timeRegions":[],"timeShift":null,"title":"vCPU","tooltip":{"shared":true,"sort":0,"value_type":"individual"},"type":"graph","xaxis":{"buckets":null,"mode":"time","name":null,"show":true,"values":[]},"yaxes":[{"$$hashKey":"object:447","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true},{"$$hashKey":"object:448","format":"short","label":null,"logBase":1,"max":null,"min":null,"show":true}],"yaxis":{"align":false,"alignLevel":null}}],"refresh":false,"schemaVersion":26,"style":"dark","tags":[],"templating":{"list":[]},"time":{"from":"now-5m","to":"now"},"timepicker":{"refresh_intervals":["10s","30s","1m","5m","15m","30m","1h","2h","1d"]},"timezone":"","title":"Istio Wasm Extension Dashboard","uid":"7PAV7ctGz","version":17} @@ -887,7 +892,4 @@ data: source_workload=~\\\"$workload\\\", source_workload_namespace=~\\\"$namespace\\\"}) by (destination_service))\",\"refresh\":1,\"regex\":\"/.*destination_service=\\\"([^\\\"]*).*/\",\"skipUrlSync\":false,\"sort\":4,\"tagValuesQuery\":\"\",\"tags\":[],\"tagsQuery\":\"\",\"type\":\"query\",\"useTags\":false}]},\"time\":{\"from\":\"now-5m\",\"to\":\"now\"},\"timepicker\":{\"refresh_intervals\":[\"5m\",\"15m\",\"30m\",\"1h\",\"2h\",\"1d\"],\"time_options\":[\"5m\",\"15m\",\"1h\",\"6h\",\"12h\",\"24h\",\"2d\",\"7d\",\"30d\"]},\"timezone\":\"\",\"title\":\"Istio Workload Dashboard\",\"uid\":\"UbsSZTDik\",\"version\":1}\n" -kind: ConfigMap -metadata: - creationTimestamp: null - name: istio-services-grafana-dashboards + diff --git a/prometheus/values.yaml b/prometheus/values.yaml index d4e72eb6..e773f13f 100644 --- a/prometheus/values.yaml +++ b/prometheus/values.yaml @@ -105,6 +105,9 @@ prometheus: replacement: $1:$2 - action: labelmap regex: __meta_kubernetes_service_label_(.+) + - source_labels: [__meta_kubernetes_namespace] + action: drop + regex: kyma-system - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace @@ -154,6 +157,9 @@ prometheus: target_label: __address__ - action: labelmap regex: __meta_kubernetes_pod_label_(.+) + - source_labels: [__meta_kubernetes_namespace] + action: drop + regex: kyma-system - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace From 3f0391fb69234f3a81101438b87dc652e1233bb7 Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Mon, 26 Sep 2022 17:21:12 +0200 Subject: [PATCH 10/30] fix --- .../deployment/deployment.yaml | 5 +++-- prometheus/README.md | 3 ++- prometheus/values.yaml | 20 ++++++++++--------- 3 files changed, 16 insertions(+), 12 deletions(-) diff --git a/monitoring-custom-metrics/deployment/deployment.yaml b/monitoring-custom-metrics/deployment/deployment.yaml index 96060f0f..276eb417 100644 --- a/monitoring-custom-metrics/deployment/deployment.yaml +++ b/monitoring-custom-metrics/deployment/deployment.yaml @@ -14,8 +14,6 @@ spec: example: monitoring-custom-metrics template: metadata: - annotations: - traffic.sidecar.istio.io/includeInboundPorts: "8080" labels: app: sample-metrics example: monitoring-custom-metrics @@ -29,6 +27,9 @@ spec: memory: 100Mi requests: memory: 32Mi + ports: + - name: http + containerPort: 8080 --- kind: Service apiVersion: v1 diff --git a/prometheus/README.md b/prometheus/README.md index 9ae91e4a..bd28a87f 100644 --- a/prometheus/README.md +++ b/prometheus/README.md @@ -23,6 +23,7 @@ Be aware of that this tutorial describes a basic setup which is not to be used i ```bash export KYMA_EXAMPLE_NS="{namespace}" ``` + >Note: Please assure that this namespace has no Istio sidecar injection enabled. The helm chart will deploy jobs which will not succeed having sidecar injection enabled by default. 2. Export the Helm release name that you want to use. It can be any name, but be aware that all resources in the cluster will be prefixed with that name. Replace the `{release-name}` placeholder in the following command and run it: ```bash @@ -92,7 +93,7 @@ Instead of defining a ServiceMonitor per workload for setting up custom metric s ```yaml prometheus.io/scrape: "true" # mandatory to enable automatic scraping -prometheus.io/scheme: https # optional, default is http. Use "https" to scrape using istio client certificates +prometheus.io/scheme: https # optional, default is http. Use "https" to scrape using istio client certificates. Will only work for services (not pods) prometheus.io/port: "1234" # optional, configure the port under which the metrics are exposed prometheus.io/path: /myMetrics # optional, configure the path under which the metrics are exposed ``` diff --git a/prometheus/values.yaml b/prometheus/values.yaml index e773f13f..668cac65 100644 --- a/prometheus/values.yaml +++ b/prometheus/values.yaml @@ -12,6 +12,12 @@ prometheus-node-exporter: service: port: 9101 targetPort: 9101 + annotations: +####### This block disables prometheus annotations on monitoring components in order to be able to enable scraping via annotation for all workloads + prometheus.io/scrape: "false" + +kube-state-metrics: + prometheusScrape: false ####### This block is disabling control plane components which are not available in the Gardener data plane @@ -35,8 +41,9 @@ kubeScheduler: prometheus: prometheusSpec: podMetadata: - annotations: + labels: sidecar.istio.io/inject: "true" + annotations: traffic.sidecar.istio.io/includeInboundPorts: "" # do not intercept any inbound ports traffic.sidecar.istio.io/includeOutboundIPRanges: "" # do not intercept any outbound traffic proxy.istio.io/config: | @@ -68,7 +75,7 @@ prometheus: ruleSelectorNilUsesHelmValues: false ####### This block enabled scrape discovery based on typical prometheus annotations - + additionalScrapeConfigs: # Scrape config for service endpoints. # @@ -107,7 +114,7 @@ prometheus: regex: __meta_kubernetes_service_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: drop - regex: kyma-system + regex: kyma-system|kube-system - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace @@ -159,15 +166,10 @@ prometheus: regex: __meta_kubernetes_pod_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: drop - regex: kyma-system + regex: kyma-system|kube-system - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_pod_name] action: replace target_label: kubernetes_pod_name - tls_config: - ca_file: /etc/prometheus/secrets/istio.default/root-cert.pem - cert_file: /etc/prometheus/secrets/istio.default/cert-chain.pem - insecure_skip_verify: true - key_file: /etc/prometheus/secrets/istio.default/key.pem From 68ee47cdf53cc0d45dae584a0d0186b89e04c98d Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Mon, 26 Sep 2022 17:32:59 +0200 Subject: [PATCH 11/30] fix --- prometheus/README.md | 2 +- prometheus/values.yaml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/prometheus/README.md b/prometheus/README.md index bd28a87f..441ec6ee 100644 --- a/prometheus/README.md +++ b/prometheus/README.md @@ -6,7 +6,7 @@ The Kyma monitoring stack often brings limited configuration options in contrast As an alternative, you can install the upstream chart with all customization options parallel. This tutorial outlines how to set up such installation in co-existence to the Kyma monitoring stack. -Be aware of that this tutorial describes a basic setup which is not to be used in production. Further configuration is usually required like optimizations in regards to the amount of data to scrape and the required resource footprint of the installation. Even a different setup might be required to achieve qualities like high availability, scalability or durable long-term storage. +> **CAUTION:** This tutorial describes a basic setup that you should not use in production. Typically, a production setup needs further configuration, like optimizing the amount of data to scrape and the required resource footprint of the installation. To achieve qualities like high availability, scalability, or durable long-term storage, you might need an even more advanced setup. ## Prerequisites diff --git a/prometheus/values.yaml b/prometheus/values.yaml index 668cac65..23beb577 100644 --- a/prometheus/values.yaml +++ b/prometheus/values.yaml @@ -19,7 +19,7 @@ prometheus-node-exporter: kube-state-metrics: prometheusScrape: false -####### This block is disabling control plane components which are not available in the Gardener data plane +####### This block disables control plane components which are not reachable from within the Gardener data plane # Disable scraping of etcd kubeEtcd: From af66a862734022ba330787c62a895ca2378ca7ef Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Mon, 26 Sep 2022 17:35:05 +0200 Subject: [PATCH 12/30] fix --- CODEOWNERS | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CODEOWNERS b/CODEOWNERS index 0b30a965..fa1fefed 100644 --- a/CODEOWNERS +++ b/CODEOWNERS @@ -8,7 +8,7 @@ * @a-thaler @lilitgh @rakesh-garimella @hisarbalik @skhalash @chrkl @lindnerby # Service binding example -/service-binding/ @Abd4llA @joek @rakesh-garimella @venturasr @lilitgh @k15r @nachtmaar @anishj0shi @montaro @abbi-gaurav @marcobebway @radufa @sayanh +/service-binding/ @rakesh-garimella @lilitgh @k15r @nachtmaar @marcobebway # Orders service example /orders-service/ @m00g3n @pPrecel From 0090823d01f6aaa7f2ecd09bb40004ab8b64985e Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Mon, 26 Sep 2022 17:36:05 +0200 Subject: [PATCH 13/30] fix --- CODEOWNERS | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/CODEOWNERS b/CODEOWNERS index fa1fefed..de0145b5 100644 --- a/CODEOWNERS +++ b/CODEOWNERS @@ -29,7 +29,7 @@ /scale-to-zero-with-kedam @m00g3n @pPrecel @dbadura @kwiatekus @moelsayed @cortey # All .md files -*.md @klaudiagrz @mmitoraj @majakurcius @alexandra-simeonova @NHingerl @grego952 +*.md @mmitoraj @majakurcius @NHingerl @grego952 # Config file for MILV - milv.config.yaml -milv.config.yaml @m00g3n @aerfio @pPrecel @magicmatatjahu +milv.config.yaml @m00g3n @pPrecel From c02f66b1ef3bf2a37a43349056780e33eea15c09 Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Wed, 28 Sep 2022 15:28:46 +0200 Subject: [PATCH 14/30] fix --- monitoring-custom-metrics/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/monitoring-custom-metrics/README.md b/monitoring-custom-metrics/README.md index fcc5de4c..cc16d9ae 100644 --- a/monitoring-custom-metrics/README.md +++ b/monitoring-custom-metrics/README.md @@ -49,7 +49,7 @@ This example shows how to expose custom metrics to Prometheus with a Golang serv All the **sample-metrics** endpoints appear as the [`Targets`](http://localhost:9090/targets#job-sample-metrics) list. 2. Use either `cpu_temperature_celsius` or `hd_errors_total` in the **expression** field [here](http://localhost:9090/graph). -3. Click the **Execute** button to check the values scrapped by Prometheus. +3. Click the **Execute** button to check the values scraped by Prometheus. ### Cleanup From 43c9510117f53eb260060db6d0a4c8256e14c08c Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Thu, 29 Sep 2022 15:34:30 +0200 Subject: [PATCH 15/30] fix --- prometheus/values.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/prometheus/values.yaml b/prometheus/values.yaml index 23beb577..1e7d9e1e 100644 --- a/prometheus/values.yaml +++ b/prometheus/values.yaml @@ -114,7 +114,7 @@ prometheus: regex: __meta_kubernetes_service_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: drop - regex: kyma-system|kube-system + regex: kyma-system|kube-system|compass-system|kyma-integration - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace @@ -166,7 +166,7 @@ prometheus: regex: __meta_kubernetes_pod_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: drop - regex: kyma-system|kube-system + regex: kyma-system|kube-system|compass-system|kyma-integration - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace From e24303ec2203103c29c98f78913ad4fd25b8236a Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Thu, 29 Sep 2022 16:42:53 +0200 Subject: [PATCH 16/30] fix --- prometheus/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/prometheus/README.md b/prometheus/README.md index 441ec6ee..d278541f 100644 --- a/prometheus/README.md +++ b/prometheus/README.md @@ -76,7 +76,7 @@ Please have in mind that this setup will collect all istio metrics on a pod leve 1. You should see several Pods coming up in the Namespace, especially Prometheus and Alertmanager. Assure that all Pods have the "Running" state. 2. Browse the Prometheus dashboard and verify that all "Status->Targets" are healthy. The following command exposes the dashboard on `http://localhost:9090`: ```bash - kubectl -n ${KYMA_EXAMPLE_NS} port-forward svc/${HELM_RELEASE_NAME}-kube-prometheus-stack-prometheus 9090 + kubectl -n ${KYMA_EXAMPLE_NS} port-forward svc/${HELM_RELEASE_NAME}-kube-prometh-prometheus 9090 ``` 3. Browse the Grafana dashboard and verify that the dashboards are showing data. The user `admin` is pre-configured in the Helm chart; the password was provided in your `helm install` command. The following command exposes the dashboard on `http://localhost:3000`: ```bash From 5eeabb9b4ea29e6b22287cf588577e494d2a087b Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Thu, 13 Oct 2022 17:08:59 +0200 Subject: [PATCH 17/30] Apply suggestions from code review Co-authored-by: Nina Hingerl <76950046+NHingerl@users.noreply.github.com> --- prometheus/README.md | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/prometheus/README.md b/prometheus/README.md index d278541f..415c8537 100644 --- a/prometheus/README.md +++ b/prometheus/README.md @@ -23,7 +23,7 @@ As an alternative, you can install the upstream chart with all customization opt ```bash export KYMA_EXAMPLE_NS="{namespace}" ``` - >Note: Please assure that this namespace has no Istio sidecar injection enabled. The helm chart will deploy jobs which will not succeed having sidecar injection enabled by default. + >Note: This Namespace must have **no** Istio sidecar injection enabled. The Helm chart deploys jobs that will not succeed when sidecar injection is enabled by default. 2. Export the Helm release name that you want to use. It can be any name, but be aware that all resources in the cluster will be prefixed with that name. Replace the `{release-name}` placeholder in the following command and run it: ```bash @@ -51,25 +51,27 @@ The provided `values.yaml` covers the following adjustments: ### Install Istio Support -To configure Prometheus for scraping of the istio specific metrics from any istio-proxy running in the cluster, deploy a PodMonitor which scrapes any pod having a port with name `.*-envoy-prom` exposed. +1. To configure Prometheus for scraping of the Istio-specific metrics from any istio-proxy running in the cluster, deploy a PodMonitor, which scrapes any Pod that has a port with name `.*-envoy-prom` exposed. ```bash kubectl -n ${KYMA_EXAMPLE_NS} apply -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/istio/podmonitor-istio-proxy.yaml ``` -Also deploy a ServiceMonitor definition for the central metrics of the `istiod` deployment. +2. Deploy a ServiceMonitor definition for the central metrics of the `istiod` deployment: ```bash kubectl -n ${KYMA_EXAMPLE_NS} apply -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/istio/servicemonitor-istiod.yaml ``` -As Grafana is configured to load dashboards dynamically from ConfigMaps in the cluster, Istio specific dashboards can be applied as well. To get the latest versions you can follow the official [instructions](https://istio.io/latest/docs/ops/integrations/grafana/#option-1-quick-start) or you take the prepared ones by calling: +3. Get the latest versions of the Istio-specific dashboards. + Grafana is configured to load dashboards dynamically from ConfigMaps in the cluster, so Istio-specific dashboards can be applied as well. + Either follow the [Istio quick start instructions](https://istio.io/latest/docs/ops/integrations/grafana/#option-1-quick-start), or take the prepared ones with the following command: ```bash kubectl -n ${KYMA_EXAMPLE_NS} apply -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/istio/configmap-istio-grafana-dashboards.yaml kubectl -n ${KYMA_EXAMPLE_NS} apply -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/istio/configmap-istio-services-grafana-dashboards.yaml ``` -Please have in mind that this setup will collect all istio metrics on a pod level. As this can lead to cardinality issues and metrics are only needed on service level, it is recommended to use an improved setup based on federation as described in the [istio documentation](https://istio.io/latest/docs/ops/best-practices/observability/#using-prometheus-for-production-scale-monitoring). + > **NOTE:** This setup collects all Istio metrics on a Pod level, which can lead to cardinality issues. Because metrics are only needed on service level, for setups having a bigger amount of workloads deployed, it is recommended to use a setup based on federation as described in the [Istio documentation](https://istio.io/latest/docs/ops/best-practices/observability/#using-prometheus-for-production-scale-monitoring). ### Verify the installation @@ -89,7 +91,7 @@ Please have in mind that this setup will collect all istio metrics on a pod leve ### Scrape workload via annotations -Instead of defining a ServiceMonitor per workload for setting up custom metric scraping, you can use a simplified way based on annotations. The used [values.yaml](./values.yaml) defines an `additionalScrapeConfig` which will scrape all pods and services having these annotations: +Instead of defining a ServiceMonitor per workload for setting up custom metric scraping, you can use a simplified way based on annotations. The used [values.yaml](./values.yaml) defines an `additionalScrapeConfig`, which scrapes all Pods and services that have these annotations: ```yaml prometheus.io/scrape: "true" # mandatory to enable automatic scraping @@ -98,13 +100,12 @@ prometheus.io/port: "1234" # optional, configure the port under which the me prometheus.io/path: /myMetrics # optional, configure the path under which the metrics are exposed ``` -You can try it out by removing the ServiceMonitor of the example used above and instead providing the annotations to the Service manifest. +You can try it out by removing the ServiceMonitor from the previous example and instead providing the annotations to the Service manifest. ### Cleanup -Run the following command to remove the installation from the cluster: +1. To remove the installation from the cluster, call Helm: -1. Remove the stack by calling Helm: ```bash helm delete -n ${KYMA_EXAMPLE_NS} ${HELM_RELEASE_NAME} From 771168fa55520c8dc049ad4d5873931089575c86 Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Thu, 13 Oct 2022 17:13:38 +0200 Subject: [PATCH 18/30] fix intendation --- prometheus/README.md | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) diff --git a/prometheus/README.md b/prometheus/README.md index 415c8537..02e4a98a 100644 --- a/prometheus/README.md +++ b/prometheus/README.md @@ -8,6 +8,8 @@ As an alternative, you can install the upstream chart with all customization opt > **CAUTION:** This tutorial describes a basic setup that you should not use in production. Typically, a production setup needs further configuration, like optimizing the amount of data to scrape and the required resource footprint of the installation. To achieve qualities like high availability, scalability, or durable long-term storage, you might need an even more advanced setup. +> **CAUTION:** This example will use the latest Grafana version which is under AGPL-3.0 and might not be free of charge for commercial usage. + ## Prerequisites - Kyma as the target deployment environment. @@ -53,23 +55,24 @@ The provided `values.yaml` covers the following adjustments: 1. To configure Prometheus for scraping of the Istio-specific metrics from any istio-proxy running in the cluster, deploy a PodMonitor, which scrapes any Pod that has a port with name `.*-envoy-prom` exposed. -```bash -kubectl -n ${KYMA_EXAMPLE_NS} apply -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/istio/podmonitor-istio-proxy.yaml -``` + ```bash + kubectl -n ${KYMA_EXAMPLE_NS} apply -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/istio/podmonitor-istio-proxy.yaml + ``` 2. Deploy a ServiceMonitor definition for the central metrics of the `istiod` deployment: -```bash -kubectl -n ${KYMA_EXAMPLE_NS} apply -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/istio/servicemonitor-istiod.yaml -``` + + ```bash + kubectl -n ${KYMA_EXAMPLE_NS} apply -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/istio/servicemonitor-istiod.yaml + ``` 3. Get the latest versions of the Istio-specific dashboards. Grafana is configured to load dashboards dynamically from ConfigMaps in the cluster, so Istio-specific dashboards can be applied as well. Either follow the [Istio quick start instructions](https://istio.io/latest/docs/ops/integrations/grafana/#option-1-quick-start), or take the prepared ones with the following command: -```bash -kubectl -n ${KYMA_EXAMPLE_NS} apply -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/istio/configmap-istio-grafana-dashboards.yaml -kubectl -n ${KYMA_EXAMPLE_NS} apply -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/istio/configmap-istio-services-grafana-dashboards.yaml -``` + ```bash + kubectl -n ${KYMA_EXAMPLE_NS} apply -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/istio/configmap-istio-grafana-dashboards.yaml + kubectl -n ${KYMA_EXAMPLE_NS} apply -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/istio/configmap-istio-services-grafana-dashboards.yaml + ``` > **NOTE:** This setup collects all Istio metrics on a Pod level, which can lead to cardinality issues. Because metrics are only needed on service level, for setups having a bigger amount of workloads deployed, it is recommended to use a setup based on federation as described in the [Istio documentation](https://istio.io/latest/docs/ops/best-practices/observability/#using-prometheus-for-production-scale-monitoring). From 2474608b2fef88ee7bd107ba944966dc03b1892b Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Thu, 13 Oct 2022 17:30:25 +0200 Subject: [PATCH 19/30] fix --- prometheus/README.md | 2 +- prometheus/values.yaml | 9 +++++---- 2 files changed, 6 insertions(+), 5 deletions(-) diff --git a/prometheus/README.md b/prometheus/README.md index 02e4a98a..2fb77149 100644 --- a/prometheus/README.md +++ b/prometheus/README.md @@ -98,7 +98,7 @@ Instead of defining a ServiceMonitor per workload for setting up custom metric s ```yaml prometheus.io/scrape: "true" # mandatory to enable automatic scraping -prometheus.io/scheme: https # optional, default is http. Use "https" to scrape using istio client certificates. Will only work for services (not pods) +prometheus.io/scheme: https # optional, default is http when no istio sidecar is used. When using a sidecar (pod has label security.istio.io/tlsMode=istio) the default will be "https". Use "https" to scrape workloads using istio client certificates. Will only work for annotated services (not pods) prometheus.io/port: "1234" # optional, configure the port under which the metrics are exposed prometheus.io/path: /myMetrics # optional, configure the path under which the metrics are exposed ``` diff --git a/prometheus/values.yaml b/prometheus/values.yaml index 1e7d9e1e..7a5083f7 100644 --- a/prometheus/values.yaml +++ b/prometheus/values.yaml @@ -97,6 +97,11 @@ prometheus: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape] action: keep regex: true + - source_labels: [__meta_kubernetes_pod_label_security_istio_io_tlsMode] + action: replace + target_label: __scheme__ + replacement: https + regex: (istio) - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme] action: replace target_label: __scheme__ @@ -149,10 +154,6 @@ prometheus: - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape] action: keep regex: true - - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme] - action: replace - target_label: __scheme__ - regex: (https?) - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ From 54dfdeb5d7f0f1635122beeada7cc192f6ef526b Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Wed, 26 Oct 2022 13:13:41 +0200 Subject: [PATCH 20/30] Update README.md --- prometheus/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/prometheus/README.md b/prometheus/README.md index 2fb77149..c59516a8 100644 --- a/prometheus/README.md +++ b/prometheus/README.md @@ -6,7 +6,7 @@ The Kyma monitoring stack often brings limited configuration options in contrast As an alternative, you can install the upstream chart with all customization options parallel. This tutorial outlines how to set up such installation in co-existence to the Kyma monitoring stack. -> **CAUTION:** This tutorial describes a basic setup that you should not use in production. Typically, a production setup needs further configuration, like optimizing the amount of data to scrape and the required resource footprint of the installation. To achieve qualities like high availability, scalability, or durable long-term storage, you might need an even more advanced setup. +> **CAUTION:** This tutorial describes a basic setup that you should not use in production. Typically, a production setup needs further configuration, like optimizing the amount of data to scrape and the required resource footprint of the installation. To achieve qualities like [high availability](https://prometheus.io/docs/introduction/faq/#can-prometheus-be-made-highly-available), [scalability](https://prometheus.io/docs/introduction/faq/#i-was-told-prometheus-doesnt-scale), or [durable long-term storage](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage), you need a more advanced setup. > **CAUTION:** This example will use the latest Grafana version which is under AGPL-3.0 and might not be free of charge for commercial usage. From d0621fe21f7e0e1b3b07e2836e1ced038288a09f Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Wed, 26 Oct 2022 13:24:13 +0200 Subject: [PATCH 21/30] Update README.md --- prometheus/README.md | 43 ++++++++++++++++++++++++++----------------- 1 file changed, 26 insertions(+), 17 deletions(-) diff --git a/prometheus/README.md b/prometheus/README.md index c59516a8..c00cb6a6 100644 --- a/prometheus/README.md +++ b/prometheus/README.md @@ -23,16 +23,25 @@ As an alternative, you can install the upstream chart with all customization opt 1. Export your Namespace as a variable. Replace the `{namespace}` placeholder in the following command and run it: ```bash - export KYMA_EXAMPLE_NS="{namespace}" + export KYMA_NS="{namespace}" + ``` +1. If you don't have it created yet, now is the time to do so: + ```bash + kubectl create namespace $KYMA_NS ``` - >Note: This Namespace must have **no** Istio sidecar injection enabled. The Helm chart deploys jobs that will not succeed when sidecar injection is enabled by default. -2. Export the Helm release name that you want to use. It can be any name, but be aware that all resources in the cluster will be prefixed with that name. Replace the `{release-name}` placeholder in the following command and run it: +1. Assure that your namespace has istio-injection disabled by having the proper label in place ```bash - export HELM_RELEASE_NAME="{release-name}" + kubectl label namespace $KYMA_NS istio-injection=disabled + ``` + >**Note**: This Namespace must have **no** Istio sidecar injection enabled. The Helm chart deploys jobs that will not succeed when sidecar injection is enabled by default. + +1. Export the Helm release name that you want to use. It can be any name, but be aware that all resources in the cluster will be prefixed with that name. Replace the `{release-name}` placeholder in the following command and run it: + ```bash + export HELM_RELEASE="{release-name}" ``` -3. Update your Helm installation with the required Helm repository: +1. Update your Helm installation with the required Helm repository: ```bash helm repo add prometheus-community https://prometheus-community.github.io/helm-charts @@ -43,26 +52,27 @@ As an alternative, you can install the upstream chart with all customization opt 1. Run the Helm upgrade command, which installs the chart if it's not present yet. At the end of the command, change the Grafana admin password to some value of your choice. ```bash - helm upgrade --install -n ${KYMA_EXAMPLE_NS} ${HELM_RELEASE_NAME} prometheus-community/kube-prometheus-stack -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/values.yaml --set grafana.adminPassword=myPwd + helm upgrade --install -n ${KYMA_NS} ${HELM_RELEASE} prometheus-community/kube-prometheus-stack -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/values.yaml --set grafana.adminPassword=myPwd ``` You can use the [values.yaml](./values.yaml) provided with this tutorial, which contains customized settings deviating from the default settings, or create your own one. The provided `values.yaml` covers the following adjustments: -- Support parallel operation to a Kyma monitoring stack -- Support the scraping of workload secured with Istio strict mTLS +- Parallel operation to a Kyma monitoring stack +- Client certificate injection to support scraping of workload secured with Istio strict mTLS +- Active scraping of workload annotated with @prometheus.io/scrape -### Install Istio Support +### Activate scraping of Istio metrics & Grafana dashboards 1. To configure Prometheus for scraping of the Istio-specific metrics from any istio-proxy running in the cluster, deploy a PodMonitor, which scrapes any Pod that has a port with name `.*-envoy-prom` exposed. ```bash - kubectl -n ${KYMA_EXAMPLE_NS} apply -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/istio/podmonitor-istio-proxy.yaml + kubectl -n ${KYMA_NS} apply -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/istio/podmonitor-istio-proxy.yaml ``` 2. Deploy a ServiceMonitor definition for the central metrics of the `istiod` deployment: ```bash - kubectl -n ${KYMA_EXAMPLE_NS} apply -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/istio/servicemonitor-istiod.yaml + kubectl -n ${KYMA_NS} apply -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/istio/servicemonitor-istiod.yaml ``` 3. Get the latest versions of the Istio-specific dashboards. @@ -70,8 +80,8 @@ The provided `values.yaml` covers the following adjustments: Either follow the [Istio quick start instructions](https://istio.io/latest/docs/ops/integrations/grafana/#option-1-quick-start), or take the prepared ones with the following command: ```bash - kubectl -n ${KYMA_EXAMPLE_NS} apply -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/istio/configmap-istio-grafana-dashboards.yaml - kubectl -n ${KYMA_EXAMPLE_NS} apply -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/istio/configmap-istio-services-grafana-dashboards.yaml + kubectl -n ${KYMA_NS} apply -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/istio/configmap-istio-grafana-dashboards.yaml + kubectl -n ${KYMA_NS} apply -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/istio/configmap-istio-services-grafana-dashboards.yaml ``` > **NOTE:** This setup collects all Istio metrics on a Pod level, which can lead to cardinality issues. Because metrics are only needed on service level, for setups having a bigger amount of workloads deployed, it is recommended to use a setup based on federation as described in the [Istio documentation](https://istio.io/latest/docs/ops/best-practices/observability/#using-prometheus-for-production-scale-monitoring). @@ -81,11 +91,11 @@ The provided `values.yaml` covers the following adjustments: 1. You should see several Pods coming up in the Namespace, especially Prometheus and Alertmanager. Assure that all Pods have the "Running" state. 2. Browse the Prometheus dashboard and verify that all "Status->Targets" are healthy. The following command exposes the dashboard on `http://localhost:9090`: ```bash - kubectl -n ${KYMA_EXAMPLE_NS} port-forward svc/${HELM_RELEASE_NAME}-kube-prometh-prometheus 9090 + kubectl -n ${KYMA_NS} port-forward svc/${HELM_RELEASE}-kube-prometh-prometheus 9090 ``` 3. Browse the Grafana dashboard and verify that the dashboards are showing data. The user `admin` is pre-configured in the Helm chart; the password was provided in your `helm install` command. The following command exposes the dashboard on `http://localhost:3000`: ```bash - kubectl -n ${KYMA_EXAMPLE_NS} port-forward svc/${HELM_RELEASE_NAME}-grafana 3000:80 + kubectl -n ${KYMA_NS} port-forward svc/${HELM_RELEASE}-grafana 3000:80 ``` ### Deploy a custom workload and scrape it @@ -109,7 +119,6 @@ You can try it out by removing the ServiceMonitor from the previous example and 1. To remove the installation from the cluster, call Helm: - ```bash - helm delete -n ${KYMA_EXAMPLE_NS} ${HELM_RELEASE_NAME} + helm delete -n ${KYMA_NS} ${HELM_RELEASE} ``` From c08439321867bbee4ff85025638e0744c78b2742 Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Wed, 26 Oct 2022 15:57:09 +0200 Subject: [PATCH 22/30] Update CODEOWNERS --- CODEOWNERS | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/CODEOWNERS b/CODEOWNERS index de0145b5..c45fef6e 100644 --- a/CODEOWNERS +++ b/CODEOWNERS @@ -21,6 +21,10 @@ /prometheus/ @a-thaler @rakesh-garimella @skhalash @chrkl @lindnerby @dennis-ge @lilitgh +/kiali/ @a-thaler @rakesh-garimella @skhalash @chrkl @lindnerby @dennis-ge @lilitgh + +/trace-demo/ @a-thaler @rakesh-garimella @skhalash @chrkl @lindnerby @dennis-ge @lilitgh + /monitoring-custom-metrics/ @a-thaler @rakesh-garimella @skhalash @chrkl @lindnerby @dennis-ge @lilitgh /monitoring-custom-alerts/ @a-thaler @rakesh-garimella @skhalash @chrkl @lindnerby @dennis-ge @lilitgh From 2c4e3c67cd508f1d1b17186e989171ca19da66ce Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Fri, 28 Oct 2022 17:43:17 +0200 Subject: [PATCH 23/30] fix --- CODEOWNERS | 6 ++---- prometheus/README.md | 12 ++++++------ 2 files changed, 8 insertions(+), 10 deletions(-) diff --git a/CODEOWNERS b/CODEOWNERS index c45fef6e..bfe6ee3e 100644 --- a/CODEOWNERS +++ b/CODEOWNERS @@ -19,14 +19,12 @@ # Custom function runtime image example /custom-serverless-runtime-image/ @m00g3n @pPrecel @dbadura @kwiatekus @moelsayed @cortey +# Observability /prometheus/ @a-thaler @rakesh-garimella @skhalash @chrkl @lindnerby @dennis-ge @lilitgh - /kiali/ @a-thaler @rakesh-garimella @skhalash @chrkl @lindnerby @dennis-ge @lilitgh - +/loki/ @a-thaler @rakesh-garimella @skhalash @chrkl @lindnerby @dennis-ge @lilitgh /trace-demo/ @a-thaler @rakesh-garimella @skhalash @chrkl @lindnerby @dennis-ge @lilitgh - /monitoring-custom-metrics/ @a-thaler @rakesh-garimella @skhalash @chrkl @lindnerby @dennis-ge @lilitgh - /monitoring-custom-alerts/ @a-thaler @rakesh-garimella @skhalash @chrkl @lindnerby @dennis-ge @lilitgh # Scaling functions to zero with KEDA diff --git a/prometheus/README.md b/prometheus/README.md index c00cb6a6..8f86830f 100644 --- a/prometheus/README.md +++ b/prometheus/README.md @@ -19,6 +19,11 @@ As an alternative, you can install the upstream chart with all customization opt ## Installation ### Preparation +1. Assure that the Kyma monitoring stack running in your cluster is limited to detection of kubernetes resources in the kyma-system namespace only, so that there can be no side-effects with the additional custom stack: +> **NOTE**: That step is only needed for clusters installed manually via the Kyma CLI + ```bash + kyma deploy --component monitoring --value monitoring.prometheusOperator.namespaces.releaseNamespace=true + ``` 1. Export your Namespace as a variable. Replace the `{namespace}` placeholder in the following command and run it: @@ -29,12 +34,7 @@ As an alternative, you can install the upstream chart with all customization opt ```bash kubectl create namespace $KYMA_NS ``` - -1. Assure that your namespace has istio-injection disabled by having the proper label in place - ```bash - kubectl label namespace $KYMA_NS istio-injection=disabled - ``` - >**Note**: This Namespace must have **no** Istio sidecar injection enabled. The Helm chart deploys jobs that will not succeed when sidecar injection is enabled by default. +>**Note**: This Namespace must have **no** Istio sidecar injection enabled so no `istio-injection` label present on the namespace. The Helm chart deploys jobs that will not succeed when sidecar injection is enabled by default. 1. Export the Helm release name that you want to use. It can be any name, but be aware that all resources in the cluster will be prefixed with that name. Replace the `{release-name}` placeholder in the following command and run it: ```bash From 7b57d4fd8e34ea79ab9a4d9160494ff70bea550c Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Fri, 28 Oct 2022 17:46:25 +0200 Subject: [PATCH 24/30] fix --- prometheus/README.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/prometheus/README.md b/prometheus/README.md index 8f86830f..ff59d623 100644 --- a/prometheus/README.md +++ b/prometheus/README.md @@ -6,9 +6,9 @@ The Kyma monitoring stack often brings limited configuration options in contrast As an alternative, you can install the upstream chart with all customization options parallel. This tutorial outlines how to set up such installation in co-existence to the Kyma monitoring stack. -> **CAUTION:** This tutorial describes a basic setup that you should not use in production. Typically, a production setup needs further configuration, like optimizing the amount of data to scrape and the required resource footprint of the installation. To achieve qualities like [high availability](https://prometheus.io/docs/introduction/faq/#can-prometheus-be-made-highly-available), [scalability](https://prometheus.io/docs/introduction/faq/#i-was-told-prometheus-doesnt-scale), or [durable long-term storage](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage), you need a more advanced setup. - -> **CAUTION:** This example will use the latest Grafana version which is under AGPL-3.0 and might not be free of charge for commercial usage. +> **CAUTION:** +- This tutorial describes a basic setup that you should not use in production. Typically, a production setup needs further configuration, like optimizing the amount of data to scrape and the required resource footprint of the installation. To achieve qualities like [high availability](https://prometheus.io/docs/introduction/faq/#can-prometheus-be-made-highly-available), [scalability](https://prometheus.io/docs/introduction/faq/#i-was-told-prometheus-doesnt-scale), or [durable long-term storage](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage), you need a more advanced setup. +- This example uses the latest Grafana version, which is under AGPL-3.0 and might not be free of charge for commercial usage. ## Prerequisites @@ -30,7 +30,7 @@ As an alternative, you can install the upstream chart with all customization opt ```bash export KYMA_NS="{namespace}" ``` -1. If you don't have it created yet, now is the time to do so: +1. If you haven't created the Namespace yet, now is the time to do so: ```bash kubectl create namespace $KYMA_NS ``` @@ -108,7 +108,7 @@ Instead of defining a ServiceMonitor per workload for setting up custom metric s ```yaml prometheus.io/scrape: "true" # mandatory to enable automatic scraping -prometheus.io/scheme: https # optional, default is http when no istio sidecar is used. When using a sidecar (pod has label security.istio.io/tlsMode=istio) the default will be "https". Use "https" to scrape workloads using istio client certificates. Will only work for annotated services (not pods) +prometheus.io/scheme: https # optional, default is "http" if no Istio sidecar is used. When using a sidecar (Pod has label `security.istio.io/tlsMode=istio`), the default is "https". Use "https" to scrape workloads using Istio client certificates. Will only work for annotated services (not Pods) prometheus.io/port: "1234" # optional, configure the port under which the metrics are exposed prometheus.io/path: /myMetrics # optional, configure the path under which the metrics are exposed ``` From 134779316a20eee5bed92a119c9bce16148d7b80 Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Wed, 2 Nov 2022 09:47:38 +0100 Subject: [PATCH 25/30] Update prometheus/README.md Co-authored-by: Rakesh Garimella --- prometheus/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/prometheus/README.md b/prometheus/README.md index ff59d623..6e70ada8 100644 --- a/prometheus/README.md +++ b/prometheus/README.md @@ -91,7 +91,7 @@ The provided `values.yaml` covers the following adjustments: 1. You should see several Pods coming up in the Namespace, especially Prometheus and Alertmanager. Assure that all Pods have the "Running" state. 2. Browse the Prometheus dashboard and verify that all "Status->Targets" are healthy. The following command exposes the dashboard on `http://localhost:9090`: ```bash - kubectl -n ${KYMA_NS} port-forward svc/${HELM_RELEASE}-kube-prometh-prometheus 9090 + kubectl -n ${KYMA_NS} port-forward $(kubectl -n ${KYMA_NS} get service -l app=kube-prometheus-stack-prometheus -oname) 9090 ``` 3. Browse the Grafana dashboard and verify that the dashboards are showing data. The user `admin` is pre-configured in the Helm chart; the password was provided in your `helm install` command. The following command exposes the dashboard on `http://localhost:3000`: ```bash From 227c825dbadebf1a450db1868a0a29ebf3acce6b Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Wed, 2 Nov 2022 14:37:53 +0100 Subject: [PATCH 26/30] Update README.md --- monitoring-custom-metrics/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/monitoring-custom-metrics/README.md b/monitoring-custom-metrics/README.md index cc16d9ae..e3989eb0 100644 --- a/monitoring-custom-metrics/README.md +++ b/monitoring-custom-metrics/README.md @@ -23,8 +23,9 @@ This example shows how to expose custom metrics to Prometheus with a Golang serv 2. Ensure that your Namespace has Istio sidecar injection enabled. This example assumes that the metrics are exposed in a strict mTLS mode: - ```` + ```bash kubectl label namespace ${KYMA_EXAMPLE_NS} istio-injection=enabled + 3. Deploy the service: From d35df3c06c533147d34b7e43ca4e033aaebcb5c6 Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Wed, 2 Nov 2022 14:40:42 +0100 Subject: [PATCH 27/30] Apply suggestions from code review Co-authored-by: Nina Hingerl <76950046+NHingerl@users.noreply.github.com> --- prometheus/README.md | 24 +++++++++++------------- 1 file changed, 11 insertions(+), 13 deletions(-) diff --git a/prometheus/README.md b/prometheus/README.md index 6e70ada8..3e552cfe 100644 --- a/prometheus/README.md +++ b/prometheus/README.md @@ -2,9 +2,9 @@ ## Overview -The Kyma monitoring stack often brings limited configuration options in contrast to the upstream [`kube-prometheus-stack`](https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack) chart. Modifications might be reset at the next upgrade cycle. +The Kyma monitoring stack brings limited configuration options in contrast to the upstream [`kube-prometheus-stack`](https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack) chart. Modifications might be reset at the next upgrade cycle. -As an alternative, you can install the upstream chart with all customization options parallel. This tutorial outlines how to set up such installation in co-existence to the Kyma monitoring stack. +As an alternative, you can install the upstream chart with all customization options in parallel. This tutorial outlines how to set up such installation in co-existence to the Kyma monitoring stack. > **CAUTION:** - This tutorial describes a basic setup that you should not use in production. Typically, a production setup needs further configuration, like optimizing the amount of data to scrape and the required resource footprint of the installation. To achieve qualities like [high availability](https://prometheus.io/docs/introduction/faq/#can-prometheus-be-made-highly-available), [scalability](https://prometheus.io/docs/introduction/faq/#i-was-told-prometheus-doesnt-scale), or [durable long-term storage](https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage), you need a more advanced setup. @@ -19,8 +19,7 @@ As an alternative, you can install the upstream chart with all customization opt ## Installation ### Preparation -1. Assure that the Kyma monitoring stack running in your cluster is limited to detection of kubernetes resources in the kyma-system namespace only, so that there can be no side-effects with the additional custom stack: -> **NOTE**: That step is only needed for clusters installed manually via the Kyma CLI +1. If you cluster was installed manually using the Kyma CLI, you must assure that the Kyma monitoring stack running in your cluster is limited to detection of Kubernetes resources only in the `kyma-system` Namespace. To rule out that there are any side effects with the additional custom stack, run: ```bash kyma deploy --component monitoring --value monitoring.prometheusOperator.namespaces.releaseNamespace=true ``` @@ -34,7 +33,7 @@ As an alternative, you can install the upstream chart with all customization opt ```bash kubectl create namespace $KYMA_NS ``` ->**Note**: This Namespace must have **no** Istio sidecar injection enabled so no `istio-injection` label present on the namespace. The Helm chart deploys jobs that will not succeed when sidecar injection is enabled by default. + >**Note**: This Namespace must have **no** Istio sidecar injection enabled; that is, there must be no `istio-injection` label present on the Namespace. The Helm chart deploys jobs that will not succeed when Isto sidecar injection is enabled. 1. Export the Helm release name that you want to use. It can be any name, but be aware that all resources in the cluster will be prefixed with that name. Replace the `{release-name}` placeholder in the following command and run it: ```bash @@ -55,7 +54,7 @@ As an alternative, you can install the upstream chart with all customization opt helm upgrade --install -n ${KYMA_NS} ${HELM_RELEASE} prometheus-community/kube-prometheus-stack -f https://raw.githubusercontent.com/kyma-project/examples/main/prometheus/values.yaml --set grafana.adminPassword=myPwd ``` -You can use the [values.yaml](./values.yaml) provided with this tutorial, which contains customized settings deviating from the default settings, or create your own one. +2. You can use the [values.yaml](./values.yaml) provided with this tutorial, which contains customized settings deviating from the default settings, or create your own one. The provided `values.yaml` covers the following adjustments: - Parallel operation to a Kyma monitoring stack - Client certificate injection to support scraping of workload secured with Istio strict mTLS @@ -93,18 +92,18 @@ The provided `values.yaml` covers the following adjustments: ```bash kubectl -n ${KYMA_NS} port-forward $(kubectl -n ${KYMA_NS} get service -l app=kube-prometheus-stack-prometheus -oname) 9090 ``` -3. Browse the Grafana dashboard and verify that the dashboards are showing data. The user `admin` is pre-configured in the Helm chart; the password was provided in your `helm install` command. The following command exposes the dashboard on `http://localhost:3000`: +3. Browse the Grafana dashboard and verify that the dashboards are showing data. The user `admin` is preconfigured in the Helm chart; the password was provided in your `helm install` command. The following command exposes the dashboard on `http://localhost:3000`: ```bash kubectl -n ${KYMA_NS} port-forward svc/${HELM_RELEASE}-grafana 3000:80 ``` ### Deploy a custom workload and scrape it -1. Follow the tutorial [monitoring-custom-metrics](./../monitoring-custom-metrics/), but use the steps above to verify that the metrics are collected. +Follow the tutorial [monitoring-custom-metrics](./../monitoring-custom-metrics/), but use the steps above to verify that the metrics are collected. ### Scrape workload via annotations -Instead of defining a ServiceMonitor per workload for setting up custom metric scraping, you can use a simplified way based on annotations. The used [values.yaml](./values.yaml) defines an `additionalScrapeConfig`, which scrapes all Pods and services that have these annotations: +Instead of defining a ServiceMonitor per workload for setting up custom metric scraping, you can use a simplified way based on annotations. The used [values.yaml](./values.yaml) defines an `additionalScrapeConfig`, which scrapes all Pods and services that have the following annotations: ```yaml prometheus.io/scrape: "true" # mandatory to enable automatic scraping @@ -117,8 +116,7 @@ You can try it out by removing the ServiceMonitor from the previous example and ### Cleanup -1. To remove the installation from the cluster, call Helm: +To remove the installation from the cluster, call Helm: - ```bash - helm delete -n ${KYMA_NS} ${HELM_RELEASE} - ``` +```bash +helm delete -n ${KYMA_NS} ${HELM_RELEASE} From e3f3ffc436cf6bc7d33013b5625bf7139c4160ac Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Wed, 2 Nov 2022 14:41:33 +0100 Subject: [PATCH 28/30] update --- prometheus/README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/prometheus/README.md b/prometheus/README.md index 3e552cfe..682f9cc0 100644 --- a/prometheus/README.md +++ b/prometheus/README.md @@ -120,3 +120,4 @@ To remove the installation from the cluster, call Helm: ```bash helm delete -n ${KYMA_NS} ${HELM_RELEASE} +``` From eaa3aed02ee1bdca7c1cac28601d68df53d035e9 Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Wed, 2 Nov 2022 15:29:55 +0100 Subject: [PATCH 29/30] update --- monitoring-custom-metrics/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/monitoring-custom-metrics/README.md b/monitoring-custom-metrics/README.md index e3989eb0..6ccb2b24 100644 --- a/monitoring-custom-metrics/README.md +++ b/monitoring-custom-metrics/README.md @@ -25,7 +25,7 @@ This example shows how to expose custom metrics to Prometheus with a Golang serv ```bash kubectl label namespace ${KYMA_EXAMPLE_NS} istio-injection=enabled - + ``` 3. Deploy the service: From 686599d28f491f159057e44915d04811461927b5 Mon Sep 17 00:00:00 2001 From: Andreas Thaler Date: Wed, 2 Nov 2022 15:30:31 +0100 Subject: [PATCH 30/30] Update prometheus/README.md --- prometheus/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/prometheus/README.md b/prometheus/README.md index 682f9cc0..62538a83 100644 --- a/prometheus/README.md +++ b/prometheus/README.md @@ -36,7 +36,7 @@ As an alternative, you can install the upstream chart with all customization opt >**Note**: This Namespace must have **no** Istio sidecar injection enabled; that is, there must be no `istio-injection` label present on the Namespace. The Helm chart deploys jobs that will not succeed when Isto sidecar injection is enabled. 1. Export the Helm release name that you want to use. It can be any name, but be aware that all resources in the cluster will be prefixed with that name. Replace the `{release-name}` placeholder in the following command and run it: - ```bash + ```bash export HELM_RELEASE="{release-name}" ```