QUIC Server - Testing - Coverage review #533
Comments
|
Run coveralls on a feature branch |
|
The current coverage of the 3.4 branch is 67.81% That is 0.08% coverage drop from the master branch and additional 0.03% coverage drop between 3.4 and master branch. We are still expecting the server-side fuzz testing to be done which can hopefully fix the drop from master. |
|
Waiting to see how quic server fuzz testing from #384 increases code coverage |
|
This is the current coverage report for the feature/quic-server branch: There are some obvious gaps. So we could write some quicapi test tests. |
|
Some interesting data: I ran a local coverage test on the mater branch, and again on the quic-server branch (adding in the quic-server fuzzer and few obvious fixes). a diff of the output of the coverage files run through lcov --list is as follows: |
|
ok, this is getting a bit strange. I've run lcov locally using the ubuntu 24.04 container (to make generation on par with what CI is doing). Results linked below In the comparison, quic-server is doing better than its counter part on the master branch overall (81.1% on quic-server vs 78.8% on master), and overall the coverage report shows that the quic-server branch has 66.6% coverage vs 66.3% on master However, neither of those values approaches what coveralls is reporting in the most recent master branch (67.752%), so to go by those values we're lagging what coveralls reports as the most recent coverage number, but I would have expected the master branch to match that value, not be .4% lower). So I'm left wondering what to do here, or how to track down additional testing needs as it seems lcov has some reliability issues. |
|
I took a look at the details. Many missed lines can be accounted for error handling. I wonder if it would make sense to annotate error handling branches in code (search for |
|
I think thats a good idea. I also think it would be worthwhile to investigate the possibility of finding a unit test framework that includes mocking so that we more directly cover those error paths when warranted. I've been experimenting with CMock, but so far it fails to understand lots of our macro usage |
|
I ran coverage using the updated workflow @quarckster created, it shows different results: That puts us about 0.1% ahead of the master branch. The obvious big gap in quic based on the above run is in quic_trace.c, as we don't run any quic test with ssl tracing enabled. We could increase the coverage here fairly easily by enabling ssl tracing in the quic_multistream_test |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Currently, based on this:
https://coveralls.io/github/openssl/openssl?branch=master
Our testing coverage hits 79.67% of our code
Note, this is based of the head of the master branch, as we do not run coverage analysis on any feature branch
Based on our policy, we can't go down in coverage between releases, and I would expect that, given the size of the feature branch, our overall coverage will likely decrease when we merge. We should be prepared for that eventuality, and the need to identify missed test paths and code additional tests between our merge date and beta freeze.
The text was updated successfully, but these errors were encountered: