TRG Suggestion - Application Testing - Code Coverage #1149
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| { | ||
| "label": "TRG 10 - Application Testing" | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,62 @@ | ||
| --- | ||
| title: TRG 10.01 - Code Coverage | ||
| sidebar_position: 1 | ||
| --- | ||
|
|
||
| | Status | Created | Post-History | | ||
| |--------|-------------|-------------------------| | ||
| | Draft | 07-Feb-2025 | Initial version created | | ||
|
|
||
| ## Why | ||
|
|
||
| Goal: To ensure that all released software components meet sufficient test coverage to guarantee quality, stability, and reliability. | ||
|
|
||
| This guideline applies to all software components and projects that are part of the Eclipse Tractus-X release process. | ||
|
|
||
| ## Description | ||
|
|
||
| ### 1. Minimum Requirements | ||
|
|
||
| 1.1. **Code Coverage Threshold** | ||
|
|
||
| - Quality Gate: The minimum threshold for code coverage (line coverage) should be **80.0%**. | ||
| - This applies to both unit tests and integration tests. | ||
|
There was a problem hiding this comment. To each or that in total they cover 80% of lines? My first point would be, that imho, there is no general accepted definition of an integration test and a unit test. A common understanding differs between technologies, as some support the creation of unit test better and others favor more complex kind of integration tests. If we talk about real "integration" tests, I do not see the need for 80% code coverage, that is only redundancy that adds no value. For example, many test cases around wrong input data can typically be tested by unit tests, like a input validation in the controller can be tested by a unit test for this controller. You can expect, that an integration test with the whole service stack does not add any value, as the controller in this test will act exactly like in the unit test case. |
||
| - Every project not specifically associated to a different Quality Gate will be associated to this one by default. | ||
|
There was a problem hiding this comment. I am not happy with this setup. The current TRGs to my understanding are more about prescribing the use of a certain technical infrastructure to manage a certain quality aspect but leaves the responsibility to achieve the quality goals with the responsible persons of the repository. Here, we leave this path by defining concrete abstract requirements which have no direct impact on the quality. The risk involved with this is always that responsibility is separated. With a construct as this, quality becomes an externally defined construct specified by KPIs. As long as I fulfil the KPIs, I am done and if something goes wrong, how can I be blamed? Keep the responsibility where it belongs, with the committers of a certain repository, they have to deliver quality. And they should get support by standardized tooling, so they do now have to invent everything from scratch. |
||
|
|
||
| 1.2. **Exceptions** | ||
|
|
||
| Certain code sections may be excluded from counting towards the code coverage percentage. For example this could apply to code sections that: | ||
|
|
||
| - Are themselves test code (no "test of tests"). | ||
| - Are autogenerated code, for example Swagger-generated API clients. | ||
| - Are configuration files with no logic to test. | ||
| - Are experimental or prototype code, for example incomplete implementations hidden behind feature toggles. | ||
| - Are boilerplate code or entity classes that follow a known, predictable pattern. | ||
| - Are code for logging, metrics, or monitoring. | ||
| - Depend on platform-specifics, for example hardware that cannot be simulated in a test. | ||
|
|
||
| All exceptions should be documented and approved by the project's committer. | ||
|
|
||
| ### 2. Analysis and Reporting | ||
|
|
||
| 2.1. **Tools for Code Coverage Measurement** | ||
|
|
||
| - Recommended tool: **SonarCloud** as provided by Elipse Tractus-X. [Reference link](https://sonarcloud.io/organizations/eclipse-tractusx/projects) | ||
|
There was a problem hiding this comment. SonarCloud does not provide any support for code coverage measurements, it only offers support to grab results from technology specific tools, like JaCoCo for Java, see https://docs.sonarsource.com/sonarqube-cloud/enriching/test-coverage/overview/. This section does not make sense. There was a problem hiding this comment. We at the industry core hub and the sdk develop in Python, and it works There was a problem hiding this comment. There was a problem hiding this comment. Hi @matbmoser , what do you mean by "it works"? The links you provided show, that code coverage is not set up properly and some action is needed. |
||
| - Alternatives: **SonarQube, JaCoCo** | ||
| - The tool used to track code coverage should be specified and documented within the respective project. | ||
|
|
||
| 2.2. **Regular Review** | ||
|
|
||
| - Code coverage should be measured by all Eclipse Tractus-X software products at least once before each release. | ||
| - Coverage measurement results should be integrated into CI/CD pipelines and automated as part of the release process. | ||
|
|
||
| ### 3. Quality Assurance | ||
|
|
||
| 3.1. **Code Review Requirements** | ||
|
|
||
| - Reviewers must ensure that new or modified code sections are sufficiently covered by test cases. | ||
| - Code changes that lower the overall coverage below the defined Quality Gate threshold may only be accepted in exceptional cases. | ||
|
|
||
| 3.2. **Risk Analysis** | ||
|
|
||
| - If the code coverage value falls below 80.0%, the potential risks should be documented and approved by the release management team. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I strongly advice against this.
In my experience code coverage thresholds ended up in people putting fake tests that covered lines but actually tested nothing.
Test culture is something that needs to be part of the team as good practice that's put in place every day in every code change, and it's up to committers to ensure this through code reviews.
An indicator that show what's the coverage on the changed lines in a PR could help the committer to drive the review in a certain direction, but it shouldn't do the review in their place.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the line on which I commented there's the "threshold" word, that to me means that if the actual value is less, the CI should throw an error and block the contribution.
If that's not what was meant, please rephrase or remove the line altogether
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I strengthen Andreas comment here. As I already mentioned in the office hours. I would, as a first step, make a guideline in the direction:
We can give a hint, that there is some statistics that show, that 80% is a turning point and further improvements do typically not pay off, so a general target of 80% is something to think about within a team, but an absolute requirement is counterproductive. I have seen tests in my past that added no value except increasing the coverage.