Add subresults support for reportportal report plugin

[seberm]

This PR adds support for tmt subresults for tmt reportportal plugin. It tries to map the subresults as child items of parent test result in a specific launch.

Please don't review this PR while it is marked as a draft.

More info:

• https://reportportal.io/docs/api/service-api/start-child-item-using-post-1

TODOs:

• Find a way to map the subresults to parent test/step reportportal launch items
• Should the import of subresults be optional via argument? Or via fmf test metadata proposed by @kkaarreell ? What should be the default behavior?
• Try to resolve status:interrupted issues for subresults in RP, why is this happening?

Blocked by:

• Multiple reportportal plugin improvements and fixes #3356
• Fix log paths for subresults loaded from tmt-report-results.yaml #3370
• Use rhts-report-result alias instead of tmt-report-result for reporting beakerlib subresults #3372

Related:

• Add support for test phases #2826
• https://issues.redhat.com/browse/TT-306

Pull Request Checklist

• implement the feature
• write the documentation
• extend the test coverage
• update the specification
• adjust plugin docstring
• modify the json schema
• mention the version
• include a release note

[kkaarreell]

I think it might be handy to control subresults import on a per test level through test fmf metadata.

[seberm]

These are just some screenshots from my testing.

BTW: @kkaarreell Do you have any idea why the subresult items have the status:interrupted tag assigned to them? I still can't figure out why is this happening.

Launch overview (parent tmt results):

Beakerlib test detail (subresults / child test items):

Details of shell test subresults when calling tmt-report-result command multiple times:

[kkaarreell]

I have tested the current code but it doesn't work (at least) for beakerlib tests. There are no logs for individual test phases (I am not even sure if beakerlib even produces them at all) and therefore there is nothing RP AA could work with or a reviewer look at when reviewing a particular failed phase. One can access logs for the whole test but because subresults are imported one can't assign a test result for a whole test as it needs to be done for individual phases. As a result, this feature is not usable ATM. As a precaution, phase reporting should be enabled only for tests that provide individual phase logs.

[seberm]

Hello @kkaarreell,
thanks for your testing and input!

I have tested the current code but it doesn't work (at least) for beakerlib tests. There are no logs for individual test phases (I am not even sure if beakerlib even produces them at all) and therefore there is nothing RP AA could work with or a reviewer look at when reviewing a particular failed phase.

I am aware the logs for subresults are not imported. I was troubleshooting this yesterday and the problem why the shell subresults (created by tmt-report-result -o <logfile> ...) and also beakerlib phases (see below) are not there should be fixed by this PR:

• Fix log paths for subresults loaded from tmt-report-results.yaml #3370

Regarding the logs for beakerlib subresults/phases, we want them, for sure. I was troubleshooting why the phase logs are not saved by tmt-report-result script. I've described the reason in this PR:

• Use rhts-report-result alias instead of tmt-report-result for reporting beakerlib subresults #3372

One can access logs for the whole test but because subresults are imported one can't assign a test result for a whole test as it needs to be done for individual phases. As a result, this feature is not usable ATM. As a precaution, phase reporting should be enabled only for tests that provide individual phase logs.

If I understand this correctly, if we always provide logs for ReportPortal child items (beakerlib phases and logs reported by calling tmt-report-result), we are fine and everything is nice and shiny. Right?

With both PRs (#3370, #3372) applied, the result is following:

Parent beakerlib result detail:

Parent beakerlib result logs:

Subresult logs (setup phase):

Subresult logs (myfail phase):

[kkaarreell]

On the last screenshot there is a test phase failure but the error log is missing.