TRG Suggestion - Application Testing - Code Coverage

[ds-hzimmer]

Suggestion for a Tractus-X Release Guideline (TRG) for Code Coverage

Description

Tractus-X Release Guideline (TRG) for static code analysis code coverage measurements, with a suggested quality gate target threshold of 80.0%.

The goal is to ensure that all released Eclipse Tractus-X software components meet sufficient test coverage to guarantee quality, stability, and reliability. The suggested threshold would at least initially not be technically enforced, i.e. not preventing build pipeline processes if coverage falls below this threshold while committers and teams are still implementing this new TRG for their software components.

The coverage should be measured and monitored using established Open Source tools, for which Eclipse Tractus-X already provides a SonarCloud installation. https://sonarcloud.io/organizations/eclipse-tractusx/projects
Other tools are also possible (two additional suggestions included).

Pull request created by issue:
eclipse-tractusx/sig-release#970

Pre-review checks

Please ensure to do as many of the following checks as possible, before asking for committer review:

• DEPENDENCIES are up-to-date. Dash license tool. Committers can open IP issues for restricted libs.
• Copyright and license header are present on all affected files

[ds-hzimmer]

Fixing markdown formatting blank lines, unordered list

[ndr-brt]

I strongly advice against this.
In my experience code coverage thresholds ended up in people putting fake tests that covered lines but actually tested nothing.
Test culture is something that needs to be part of the team as good practice that's put in place every day in every code change, and it's up to committers to ensure this through code reviews.

An indicator that show what's the coverage on the changed lines in a PR could help the committer to drive the review in a certain direction, but it shouldn't do the review in their place.

[ds-hzimmer]

Also note that as I mentioned in the description, we would suggest - at least initially - to only require that a code coverage measurement is implemented. With the value that is measured as an indicator, like you wrote. So not to enforce a certain threshold like 80% to prevent build/deployment processes for applications that fall below it immediately (which I have also experienced in other projects, which is obviously frustrating). We want to have code coverage as a well-established measurement, but not burden the committers / teams with having to bring up their coverage within a short period of time to meet that threshold and thereby risk they are just "faking" it.

the line on which I commented there's the "threshold" word, that to me means that if the actual value is less, the CI should throw an error and block the contribution.
If that's not what was meant, please rephrase or remove the line altogether

[lgblaumeiser]

I strengthen Andreas comment here. As I already mentioned in the office hours. I would, as a first step, make a guideline in the direction:

• Code Coverage measurements have to be in place
• Code Coverage delta reports for PRs are part of the standard PR feedback
• Committers are responsible to keep an eye on code coverage as part of their PR review

We can give a hint, that there is some statistics that show, that 80% is a turning point and further improvements do typically not pay off, so a general target of 80% is something to think about within a team, but an absolute requirement is counterproductive. I have seen tests in my past that added no value except increasing the coverage.

[florianrusch]

As just discussed at the committers meeting, I would appreciate it if we could add some examples of how to modify the pipelines to the TRG.

[lgblaumeiser]

I strengthen Andreas comment here. As I already mentioned in the office hours. I would, as a first step, make a guideline in the direction:

• Code Coverage measurements have to be in place
• Code Coverage delta reports for PRs are part of the standard PR feedback
• Committers are responsible to keep an eye on code coverage as part of their PR review

We can give a hint, that there is some statistics that show, that 80% is a turning point and further improvements do typically not pay off, so a general target of 80% is something to think about within a team, but an absolute requirement is counterproductive. I have seen tests in my past that added no value except increasing the coverage.

[lgblaumeiser]

To each or that in total they cover 80% of lines?

My first point would be, that imho, there is no general accepted definition of an integration test and a unit test. A common understanding differs between technologies, as some support the creation of unit test better and others favor more complex kind of integration tests.

If we talk about real "integration" tests, I do not see the need for 80% code coverage, that is only redundancy that adds no value. For example, many test cases around wrong input data can typically be tested by unit tests, like a input validation in the controller can be tested by a unit test for this controller. You can expect, that an integration test with the whole service stack does not add any value, as the controller in this test will act exactly like in the unit test case.

[lgblaumeiser]

I am not happy with this setup. The current TRGs to my understanding are more about prescribing the use of a certain technical infrastructure to manage a certain quality aspect but leaves the responsibility to achieve the quality goals with the responsible persons of the repository. Here, we leave this path by defining concrete abstract requirements which have no direct impact on the quality. The risk involved with this is always that responsibility is separated. With a construct as this, quality becomes an externally defined construct specified by KPIs. As long as I fulfil the KPIs, I am done and if something goes wrong, how can I be blamed? Keep the responsibility where it belongs, with the committers of a certain repository, they have to deliver quality. And they should get support by standardized tooling, so they do now have to invent everything from scratch.

[lgblaumeiser]

SonarCloud does not provide any support for code coverage measurements, it only offers support to grab results from technology specific tools, like JaCoCo for Java, see https://docs.sonarsource.com/sonarqube-cloud/enriching/test-coverage/overview/. This section does not make sense.

[matbmoser]

We at the industry core hub and the sdk develop in Python, and it works

[matbmoser]

https://sonarcloud.io/summary/new_code?id=eclipse-tractusx_tractusx-sdk&branch=main

https://sonarcloud.io/project/overview?id=eclipse-tractusx_industry-core-hub

[lgblaumeiser]

Hi @matbmoser , what do you mean by "it works"? The links you provided show, that code coverage is not set up properly and some action is needed.

[matbmoser]

We have exposed this PR in the committer meeting from 07.03.2025 and we have considered that there are some missing points where you shall investigate and add to the TRG proposal before it gets approved.

@ds-hzimmer please consider the following points:

-> Test Coverage is not available for everything.
-> Example integration and workflow shall be defined.
-> We do not want to overwhelm the eclipse foundation by creating multiple requests that are not aligned.
-> Template for the eclipse foundation so what is needed for them to set up this TRG and make it work for us, there is one secret at organization level to be created, from the CI integration the ones that created the TRG shall look in to the.