Include endtoend testing in code coverage reports

[mattlord]

We added unit test code coverage reporting in #14333.

Our unit test code coverage is currently less than 50% and we have an open issue where we will attempt to improve that: #14931.

Much of our testing today, however, is done via endtoend (e2e) tests — as we rely so heavily on RPCs — and our code coverage numbers do not currently reflect any of those.

Ideally our full code coverage report would include all of our tests. Doing that for the e2e tests will be a bigger challenge, however, as e.g. we'll have to generate a code coverage profile in each e2e test GitHub action workflow, then upload that to an s3 bucket or other shared storage. Then we run the code coverage workflow — only after the other tests have passed and after the unit test work completes — it downloads all of the e2e test coverage profiles and merges them into the one generated by the unit tests. Then run go tool cover on that aggregated profile.

The new yaml declarations which would instruct codecov to wait for all other tests to pass would be:

codecov:
  require_ci_to_pass: yes
  notify:
    wait_for_ci: yes

[glokta1]

Hi @mattlord, I wanted to take a crack at this but had a little doubt:
At what stage is make generate_ci_workflows being run to generate the e2e tests yml files? Is this happening in the CI/CD pipeline?

[GuptaManan100]

make generate_ci_workflows is something we run when we change something in the code that generates them. The files are already generated and stored in the .github/workflows directory and their generation itself isn't part of the CI/CD pipeline.

[systay]

just to circle back the discussion we've been having on slack here

seems like the codecov is not reporting these numbers accurately. according to this report, the coverage for some packages is reported as 0% because their unit tests live in a different package. (example: go/vt/vtgate/planbuilder/operators is reported to have no test coverage since it is tested from go/vt/vtgate/planbuilder/)

we're working on better numbers