Adding in user email and comparison config for kube-burner

[paigerube14]

As part of PR we added the ability to get benchmark-comparison google sheet output from kube-burner workloads. Need to add setting of comparison_config, google sheet service account and email to send it to

This only works on jenkins agents that start with "oc4*" for now. Still trying to fix for rhel8 agents

[paigerube14]

/assign @qiliRedHat

[qiliRedHat]

@paigerube14
I tested with job https://mastern-jenkins-csb-openshift-qe.apps.ocp-c1.prod.psi.redhat.com/job/scale-ci/job/paige-e2e-multibranch/job/kube-burner-3/56/console

1. There were some error msgs like

05-24 17:52:05.745  2022-05-24 09:52:05,425 - touchstone - ERROR - Error: Issue capturing results from elasticsearch using config {'filter': {'metricName.keyword': 'podCPU', 'labels.namespace.keyword': 'openshift-ovn-kubernetes'}, 'buckets': ['labels.pod.keyword', 'labels.node.keyword'], 'aggregations': {'value': ['avg']}}

2. From the result sheet, the last 2 columns data were at wrong places.

OCP Version Flexy Id Scale Ci Job UUID/Grafana URL Status Params Pod Ready Average (ms) Pod Scheduled Avg (ms) Initialized (ms) Containers Ready (ms) Time/Date Snapshot (done manually)

4.11.0-0.nightly-2022-05-18-171831 104086 56 15a42470-aa6b-4f45-863f-3bc171d9156a PASS 1 2405 0 42 2405 https://docs.google.com/spreadsheets/d/1NHU-mD1MqZcZizeJkKvAhpvXVKpKb3ogMEj70qjbhcs *************** 2022-05-24 04:53:58.448785-05:00

[paigerube14]

@qiliRedHat those errors need to be fixed upstream I believe. I will run it locally with the same parameters and report back with the results

[paigerube14]

If we set the metrics profile to be metrics-profiles/metrics-ovn.yaml it will properly get the missing metrics and runs cleaner

Error message before setting metrics-ovn.yaml file
touchstone - ERROR - Error: Issue capturing results from elasticsearch using config {'filter': {'metricName.keyword': 'podMemory', 'labels.namespace.keyword': 'openshift-sdn'}, 'buckets': ['labels.pod.keyword', 'labels.node.keyword'], 'aggregations': {'value': ['max']}}

Successful run: https://mastern-jenkins-csb-openshift-qe.apps.ocp-c1.prod.psi.redhat.com/job/scale-ci/job/paige-e2e-multibranch/job/kube-burner-3/63/

@qiliRedHat what are your thoughts on setting the default metrics profile to metrics-ovn.yaml for our runs? The 2 files are the exact same except for the addition of podCPU and podMemory metrics are indexed using the ovn yaml file and not the metrics.yaml file

https://github.com/cloud-bulldozer/e2e-benchmarking/blob/4fa5a5ce4856bdc309cba8bab41ed4b68c251c97/workloads/kube-burner/metrics-profiles/metrics-ovn.yaml#L20

[paigerube14]

Created PR to not have to specifically set the metrics-ovn.yaml file as the metrics profile so we can properly get the podCPU and podMemory metrics in our google sheet comparison file

[qiliRedHat]

@paigerube14 My run's google sheet is shared to you https://docs.google.com/spreadsheets/d/1NHU-mD1MqZcZizeJkKvAhpvXVKpKb3ogMEj70qjbhcs/edit?usp=sharing.
ContainersReady/Initialized/PodScheduled/Ready's P99 data in podLatency.json are all 0. Are you seeing the same? I asked the access to your run's google sheet.

[paigerube14]

@qiliRedHat no I have not seen that. Looks like the data is in elastic search for that run with 0 as the values. I have seen that nothing gets printed to the sheet though. Will have to test more

[paigerube14]

Node-density example here: https://mastern-jenkins-csb-openshift-qe.apps.ocp-c1.prod.psi.redhat.com/job/scale-ci/job/paige-e2e-multibranch/job/kube-burner-3/69/console

[qiliRedHat]

/lgtm

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: paigerube14, qiliRedHat

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [paigerube14,qiliRedHat]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment