Skip to content

Commit

Permalink
Merge pull request #18 from cloudnative-coop/updated-org
Browse files Browse the repository at this point in the history 
clear results from this org file
  • Loading branch information
@zachmandeville
zachmandeville authored Jun 6, 2023
2 parents 1c5da7e + 39b9e7b commit ee73c3c
Showing 1 changed file with 0 additions and 259 deletions.
259 changes: 0 additions & 259 deletions infrasnoop/org/ii.org
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,22 +14,6 @@ server and db, accept the defaults each time by hitting enter.
select * from describe_relations();
#+end_src

#+RESULTS:
#+begin_example
schema | name | description
--------+----------------+-------------------------------------------------------------
sigs | committee | each committee in the kubernetes sigs.yaml
sigs | sig | each sig in the kubernetes sigs.yaml
sigs | user_group | each usergroup in the kubernetes sigs.yaml
sigs | working_group | each working group in the kubernetes sigs.yaml
prow | job_annotation | every annotation of a job take from the prowspec of the job
prow | job_label | every label of a job take from the prowspec of the job
prow | job_spec | the spec from a prowjob.json expanded into sql columns
prow | latest_success | The most recent successful build of each job in prow.deck
(8 rows)

#+end_example


** Important! Load up the data
The query shows us the tables and views available in our two schemas "sigs" and "prow". Neither of these have data yet, but we can load them up
Expand All @@ -53,13 +37,6 @@ the raw data in the table ~prow.job~.
select count(*) from prow.job;
#+end_src

#+RESULTS:
: count
: -------
: 1219
: (1 row)
:

You can run the above code block multiple times and you'll see the count go up
as the sideloader works. When it is no longer adding new jobs, the database is
ready to explore.
Expand All @@ -72,15 +49,6 @@ Let's look at the prow.job table
select * from describe_columns('prow','job');
#+end_src

#+RESULTS:
: column | description
: ----------+----------------------------------------------------
: job | The prow job title. May appear multiple times.
: build_id | The exact build of this job.
: data | the prowjob definition, literally its prowjob.json
: (3 rows)
:

It is simple, with the main interestin being the data column. This is a jsonb column, So any [[https://duckduckgo.com/?t=ffab&q=postgres+operator&ia=web][postgres jsonb operator]] can be used to explore it.

In addition, we've taken some of the relevant parts of the job and turned them into their own views: prow.job_spec, prow.job_label, and prow.job_annotation.
Expand All @@ -90,53 +58,15 @@ the spec is taken from the spec key in the prow.job
\d prow.job_spec;
#+end_src

#+RESULTS:
#+begin_example
View "prow.job_spec"
Column | Type | Collation | Nullable | Default
-------------------+-------+-----------+----------+---------
job | text | | |
refs | jsonb | | |
type | jsonb | | |
agent | jsonb | | |
report | jsonb | | |
cluster | jsonb | | |
context | jsonb | | |
pod_spec | jsonb | | |
namespace | jsonb | | |
rerun_command | jsonb | | |
prowjob_defaults | jsonb | | |
decoration_config | jsonb | | |

#+end_example

while the label and annotations is taken from metadata.labels and metadata.annotations, respectively.
#+begin_src sql-mode
\d prow.job_label;
#+end_src

#+RESULTS:
: View "prow.job_label"
: Column | Type | Collation | Nullable | Default
: ---------+------+-----------+----------+---------
: job | text | | |
: label | text | | |
: content | text | | |
:

#+begin_src sql-mode
\d prow.job_annotation;
#+end_src

#+RESULTS:
: View "prow.job_annotation"
: Column | Type | Collation | Nullable | Default
: ------------+-------+-----------+----------+---------
: job | text | | |
: annotation | text | | |
: content | jsonb | | |
:

* Example prow queries
** A sanity check
First, let's look at a prow job that we know should exist in our successful jobs: the apisnoop conformance-gate.
Expand All @@ -147,13 +77,6 @@ select job
where job ilike '%apisnoop%conformance%';
#+end_src

#+RESULTS:
: job
: ---------------------------
: apisnoop-conformance-gate
: (1 row)
:

And we can see when this job ran.

#+begin_src sql-mode
Expand All @@ -163,13 +86,6 @@ select job.job, started, finished
where job.job = 'apisnoop-conformance-gate';
#+end_src

#+RESULTS:
: job | started | finished
: ---------------------------+---------------------+---------------------
: apisnoop-conformance-gate | 2023-06-04 00:46:35 | 2023-06-04 00:49:31
: (1 row)
:

And some details on its spec.

#+begin_src sql-mode
Expand All @@ -178,13 +94,6 @@ select job, type, agent, cluster, prowjob_defaults, namespace
where job = 'apisnoop-conformance-gate';
#+end_src

#+RESULTS:
: job | type | agent | cluster | prowjob_defaults | namespace
: ---------------------------+------------+--------------+-----------+----------------------------------+-------------
: apisnoop-conformance-gate | "periodic" | "kubernetes" | "default" | {"tenant_id": "GlobalDefaultID"} | "test-pods"
: (1 row)
:


We can look at its labels
#+begin_src sql-mode
Expand All @@ -193,20 +102,6 @@ select label,content
where job = 'apisnoop-conformance-gate';
#+end_src

#+RESULTS:
#+begin_example
label | content
----------------------+--------------------------------------
prow.k8s.io/id | 7c20c4c2-e061-45ff-93fd-dfb1646c8f64
created-by-prow | true
prow.k8s.io/job | apisnoop-conformance-gate
prow.k8s.io/type | periodic
prow.k8s.io/context |
prow.k8s.io/build-id | 1665158070679900160
(6 rows)

#+end_example

And its annotations:

#+begin_src sql-mode
Expand All @@ -215,21 +110,6 @@ select annotation, content
where job = 'apisnoop-conformance-gate';
#+end_src

#+RESULTS:
#+begin_example
annotation | content
--------------------------------+----------------------------------------------------------------------------------------------
description | "Uses APISnoop to check that new GA endpoints are conformance tested in latest e2e test run"
prow.k8s.io/job | "apisnoop-conformance-gate"
testgrid-tab-name | "apisnoop-conformance-gate"
prow.k8s.io/context | ""
testgrid-dashboards | "sig-arch-conformance"
test-grid-alert-email | "[email protected]"
testgrid-num-failures-to-alert | "1"
(7 rows)

#+end_example

If all the above queries returned results, then our db is set up and connected and good. Now we can do some more interesting queries.

** Jobs without a cluster
Expand All @@ -242,15 +122,6 @@ select job
where spec.cluster is null;
#+end_src

#+RESULTS:
: job
: ------------------------------------------------
: ci-kubernetes-kind-e2e-json-logging-eks-canary
: ci-containerd-build-1-6
: ar-to-s3-sync
: (3 rows)
:

When I last ran it, I got 3 results returned.

** Jobs with dind-enabled
Expand All @@ -262,13 +133,6 @@ select count(job)
where label = 'preset-dind-enabled';
#+end_src

#+RESULTS:
: count
: -------
: 516
: (1 row)
:

This will be many more, likely, so I just asked for the count.

you can always dive deeper by looking at the results, but limit to 25 rows or some other limit.
Expand All @@ -293,39 +157,6 @@ select job, label, content
limit 25 ;
#+end_src

#+RESULTS:
#+begin_example
job | label | content
----------------------------------------+-----------------------------------+----------------------------------------
build-win-soak-test-cluster | created-by-prow | true
build-win-soak-test-cluster | preset-azure-anonymous-pull | true
build-win-soak-test-cluster | preset-azure-cred-only | true
build-win-soak-test-cluster | preset-capz-containerd-1-7-latest | true
build-win-soak-test-cluster | preset-dind-enabled | true
build-win-soak-test-cluster | preset-kind-volume-mounts | true
build-win-soak-test-cluster | preset-service-account | true
build-win-soak-test-cluster | prow.k8s.io/build-id | 1665161594117558272
build-win-soak-test-cluster | prow.k8s.io/context |
build-win-soak-test-cluster | prow.k8s.io/id | 67174464-349b-464b-a406-626ebbaff5b4
build-win-soak-test-cluster | prow.k8s.io/job | build-win-soak-test-cluster
build-win-soak-test-cluster | prow.k8s.io/refs.base_ref | main
build-win-soak-test-cluster | prow.k8s.io/refs.org | kubernetes-sigs
build-win-soak-test-cluster | prow.k8s.io/refs.repo | cluster-api-provider-azure
build-win-soak-test-cluster | prow.k8s.io/type | periodic
canary-e2e-gce-cloud-provider-disabled | created-by-prow | true
canary-e2e-gce-cloud-provider-disabled | preset-dind-enabled | true
canary-e2e-gce-cloud-provider-disabled | preset-k8s-ssh | true
canary-e2e-gce-cloud-provider-disabled | preset-pull-kubernetes-e2e | true
canary-e2e-gce-cloud-provider-disabled | preset-pull-kubernetes-e2e-gce | true
canary-e2e-gce-cloud-provider-disabled | preset-service-account | true
canary-e2e-gce-cloud-provider-disabled | prow.k8s.io/build-id | 1665317371880935424
canary-e2e-gce-cloud-provider-disabled | prow.k8s.io/context |
canary-e2e-gce-cloud-provider-disabled | prow.k8s.io/id | b444ded7-3672-4631-b8fd-660b3786ba43
canary-e2e-gce-cloud-provider-disabled | prow.k8s.io/job | canary-e2e-gce-cloud-provider-disabled
(25 rows)

#+end_example

We can do a frequency count for the types of labels (note: this prolly isn't an interesting query, but useful for showing some sql tricks)

#+begin_src sql-mode
Expand All @@ -343,64 +174,6 @@ select distinct label, count(distinct job)
limit 50;
#+end_src

#+RESULTS:
#+begin_example
label | count
------------------------------------------------+-------
created-by-prow | 516
preset-dind-enabled | 516
prow.k8s.io/build-id | 516
prow.k8s.io/context | 516
prow.k8s.io/id | 516
prow.k8s.io/job | 516
prow.k8s.io/type | 516
prow.k8s.io/refs.base_ref | 505
prow.k8s.io/refs.org | 505
prow.k8s.io/refs.repo | 505
preset-kind-volume-mounts | 337
preset-service-account | 239
event-GUID | 159
prow.k8s.io/refs.pull | 159
prow.k8s.io/is-optional | 152
preset-azure-cred-only | 90
preset-azure-anonymous-pull | 79
preset-aws-credential | 54
preset-aws-ssh | 54
preset-k8s-ssh | 47
preset-azure-cred | 21
preset-azure-capz-sa-cred | 16
preset-aws-credential-aws-oss-testing | 12
preset-pull-kubernetes-e2e | 11
preset-pull-kubernetes-e2e-gce | 11
preset-capz-containerd-1-7-latest | 10
preset-e2e-kubemark-common | 10
preset-e2e-scalability-periodics | 9
preset-capz-windows-common | 8
preset-windows-private-registry-cred | 8
created-by-tide | 7
preset-azure-secrets-store-creds | 6
preset-capz-windows-2019 | 6
prow.k8s.io/retest | 6
preset-capz-serial-slow | 5
preset-capz-windows-parallel | 5
preset-e2e-scalability-periodics-master | 5
preset-azure-windows | 4
preset-capz-containerd-1-6-latest | 4
preset-cluster-api-provider-vsphere-e2e-config | 4
preset-do-credential | 4
preset-cluster-api-provider-vsphere-gcs-creds | 3
preset-bazel-remote-cache-enabled | 2
preset-capz-gmsa-setup | 2
preset-capz-windows-common-124 | 2
preset-e2e-kubemark-gce-scale | 2
preset-capz-windows-2022 | 1
preset-capz-windows-common-125 | 1
preset-capz-windows-common-126 | 1
preset-capz-windows-common-127 | 1
(50 rows)

#+end_example


* Looking into the jobs without a cluster
Before, we found the cluster-free jobs with
Expand All @@ -412,15 +185,6 @@ select job
group by job;
#+end_src

#+RESULTS:
: job
: ------------------------------------------------
: ar-to-s3-sync
: ci-containerd-build-1-6
: ci-kubernetes-kind-e2e-json-logging-eks-canary
: (3 rows)
:

this returns some # of jobs(usually 3-9)


Expand All @@ -433,15 +197,6 @@ select job, data
where spec.cluster is null;
#+end_src

#+RESULTS:
: job | data
: ------------------------------------------------+--------------------------------------------------------------------------------------------------
: ci-kubernetes-kind-e2e-json-logging-eks-canary | {"ProwJob not found": "prowjobs.prow.k8s.io \"1909bbb2-3dcc-407b-af0e-5e999a1a4b12\" not found"}
: ci-containerd-build-1-6 | {"ProwJob not found": "prowjobs.prow.k8s.io \"8e0795f0-d870-465a-9c57-ed225744af5b\" not found"}
: ar-to-s3-sync | {"ProwJob not found": "prowjobs.prow.k8s.io \"dbacec7e-43cc-4489-b398-da40ad154eb6\" not found"}
: (3 rows)
:

It's spec is just the note "ProwJob not found".

This bit of json is being pulled direct from their spyglass link, which we can grab with the below query
Expand All @@ -466,20 +221,6 @@ select cluster, count(*)
order by count desc;
#+end_src

#+RESULTS:
#+begin_example
cluster | count
--------------------------------+-------
"default" | 824
"k8s-infra-prow-build" | 216
"eks-prow-build-cluster" | 91
"k8s-infra-prow-build-trusted" | 72
"test-infra-trusted" | 13
| 3
(6 rows)

#+end_example

Is there any pattern that connects these jobs without prowjobs?

* Footnotes
Expand Down

0 comments on commit ee73c3c

Please sign in to comment.