Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Combine metric profiles #438

Closed
wants to merge 3 commits into from

Conversation

rsevilla87
Copy link
Member

Description

Combine metrics-aggregated.yaml and metrics.yaml files.

I managed to combine them thanks to expressions such as:

irate(process_cpu_seconds_total{service="kubelet",job="kubelet"}[2m]) * 100 and on (node) topk(3,avg_over_time(irate(process_cpu_seconds_total{service="kubelet",job="kubelet"}[2m])[{{ .elapsed }}:]) and on (node) kube_node_role{role="worker"})

Which uses the magic variable {{ .elapsed }} which takes the value of the benchmark duration. Now, in large-scale deployments we won't collect metrics for all worker nodes (which is overkill) but instead we'll collect metrics for the 3 worker nodes with more kubelet CPU usage, same for other similar metrics.

Signed-off-by: Raul Sevilla <[email protected]>
Signed-off-by: Raul Sevilla <[email protected]>
@rsevilla87 rsevilla87 added the ok to test Kick off our CI framework label Jun 29, 2022
@rsevilla87 rsevilla87 changed the title Combine metrics Combine metric profiles Jun 29, 2022
@rsevilla87
Copy link
Member Author

/rerun all

@comet-perf-ci
Copy link
Collaborator

Results for e2e-benchmarking CI Tests

Workload Test Result Runtime
router-perf-v2 ./ingress-performance.sh PASS 00:05:56
etcd-perf ./run_etcd_tests_fromgit.sh PASS 00:02:29
scale-perf ./run_scale_fromgit.sh PASS 00:14:21
network-perf ./run.sh PASS 00:07:20
kube-burner WORKLOAD=cluster-density ./run.sh PASS 00:05:20
kube-burner WORKLOAD=node-density ./run.sh FAIL 00:27:20
kube-burner WORKLOAD=node-density-heavy ./run.sh PASS 01:04:43
kube-burner WORKLOAD=pod-density-heavy ./run.sh PASS 00:06:10
kube-burner WORKLOAD=max-services ./run.sh PASS 00:04:51
kube-burner WORKLOAD=max-namespaces ./run.sh PASS 00:05:45
upgrade-perf ./run_upgrade_fromgit.sh PASS 00:01:06

@rsevilla87
Copy link
Member Author

/rerun all WORKLOAD=node-density ./run.sh

@comet-perf-ci
Copy link
Collaborator

Results for e2e-benchmarking CI Tests

Workload Test Result Runtime

@rsevilla87
Copy link
Member Author

/rerun all kube-burner:WORKLOAD=node-density ./run.sh

@comet-perf-ci
Copy link
Collaborator

Results for e2e-benchmarking CI Tests

Workload Test Result Runtime
kube-burner WORKLOAD=node-density ./run.sh PASS 00:05:29

@Harshith-umesh
Copy link
Member

Harshith-umesh commented Jul 1, 2022

@rsevilla87 thoughts on reviewing and updating this PR's logic #411 here? or closing it if it's no longer relevant?

@rsevilla87
Copy link
Member Author

/rerun all kube-burner:WORKLOAD=node-density ./run.sh

@comet-perf-ci
Copy link
Collaborator

Results for e2e-benchmarking CI Tests

Workload Test Result Runtime
kube-burner WORKLOAD=node-density ./run.sh PASS 00:05:25

Copy link
Member

@Harshith-umesh Harshith-umesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@rsevilla87
Copy link
Member Author

Let's put this one on hold, I have to make some change in the grafana report to ensure we don't loss any interesting metric

@rsevilla87 rsevilla87 marked this pull request as draft August 17, 2022 08:47
@rsevilla87 rsevilla87 closed this Apr 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ok to test Kick off our CI framework
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants