Discover test from sources

[lukaszachy]

Use cases looks simple - Discover tests which are included in the source of a component.

We have #588, #585 and #742 opened for this or related changes.

The extraction part can be easy enough - Get url to source tarball (based on fedora/centos/rhel and content of sources file) and extract it into provided or default location.

But after that it is a can of worms:

1. Need run '%patch' step from spec file since tests might be modified (cons: insecure for tmt executor as anything can be in spec)
2. Missing build dependencies for '%patch' step (cons: additional packages for tmt executor)
3. Wrong conditions for '%patch' (e.g. fedora vs rhel - would depend on tmt executor)
4. Ad-hoc changes to extracted sources (e.g. source omits .fmf/version) (cons: insecure for tmt executor)

I started #742 as Discover plugin but I realize that pre-discover option might be much better because:

• No need to modify discover plugins
• More implementation possibilities for pre-discover step (mock, podman, VM ...)
• Better granularity for what is allowed in pre-discover step

I'd like to have your opinion about following plan:
As a MDP setup options for discover (name subject to change) which:

• Runs on local
• Doesn't execute anything (no rpmbuild, no custom shell)
• Extracts source tarball to location for discover
• Copies spec and patches to location for discover

This will work for components which don't patch tests in spec file.
Since spec and patches will be copied too, one can run those in Prepare step as a workaround (not encouraged but possible).

As a next steps (tmt 1.9, 2.x) we provide podman/mock based setup which will execute rpm-build/install deps and so on.

MDP configuration could be

discover:
  how: fmf
  setup: 
     how: sources

Full configuration possibly (since shell is present it should be run by secure implementation or local one with explicit flag to allow shell/execution).

discover:
  how: fmf
  setup:
    how: sources
    buildrequires: auto
    target_distro: auto
    shell: "rm -rf $SOURCES/tests/destructive"

Will it work for you? Feedback please: @psss @jscotka @pvalena @thrix @jkonecny12 @martinpitt @kkaarreell

[kkaarreell]

Hi @lukaszachy , thanks for looking into this.
I wouldn't call it can of worms. It is users decision where (and under which account) he runs tests. Similarly to package building, %spec step would be also executed. If you want to be sure, you should have a dedicated system/account/container doing that.
One thing to realize, there can be multiple archives present (multiple SOURCE[0-9]+ references in SPEC file) so the question is who to reliably identify the archive to use. Unless you want to re-implement that functionality from rpmbuild we should consider using rpmbuild for unpacking sources (probably rpmbuild -bp -nodeps)

[lukaszachy]

I'd prefer to use rpmbuild.

[thrix]

@lachmanfrantisek @TomasTomecek FTR, seems in tmt we are getting into business of rpmbuild -bp, sharing in case you can provide us some guidance ...

[TomasTomecek]

/me runs away...

(btw we already have plenty of code in packit for source-git to utilize rpmbuild, extract sources and many more operations)

[lukaszachy]

Where is the source code of packit? To get the "inspiration".

[TomasTomecek]

source extraction: https://github.com/packit/packit/blob/main/packit/source_git.py#L106

fetching code from lookaside cache: https://github.com/packit/packit/blob/main/packit/source_git.py#L142

dedicated class which wraps rpmbuild functionality: https://github.com/packit/packit/blob/c02d3d553879ddef60976903bca00daa2513d9b0/packit/upstream.py#L729

[martinpitt]

@lukaszachy : I like this proposal: It does not intrude on the structure and syntax (just how: fmf → how: sources essentially, right?), and if that uses the standard RPM command to unpack the srpm (like -bp) that seems great -- that's what a local developer would do as well in other occasions. Adding arbitrary shell commands there is indeed a bit dubious security-wise, but in practice you can do bad stuff with a crafted srpm already, so 🤷‍♂️. (FTR: I don't need this, I'm fully happy to do plain sources without patching, and using fmf with a git SHA if I need to change stuff).

I am just a bit unclear what you mean by "pre-discover" step. Was that an option you discarded, or is this meant to run before the how: fmf plugin?

[jkonecny12]

We are in the same situation. We don't need patching just how: fmf -> how: sources is good enough for us. +1 for this solution.

[kkaarreell]

I might be a bit confused after reading this proposal again after couple of months but I find 'how: sources' not being very descriptive in explaining where exactly the bits are taken from. The main idea behind fmf was to keep metadata with code so stating 'sources' becomes both obvious and cryptic.

[lukaszachy]

I call it pre-discover because it runs before discover step - it prepares cwd for discover plugins. To some extend how: fmf with url specified is a combination of git clone and run discover on its results. I might be over thinking this but I'd prefer shared feature between discover plugins to each of them implementing similar functionality. Now we have just two (fmf and shell) but I hope that later tmt could discover common test frameworks (e.g. pytest) and for that reason 'sources' should be available to all discover plugins.

[lukaszachy]

I might be a bit confused after reading this proposal again after couple of months but I find 'how: sources' not being very descriptive in explaining where exactly the bits are taken from. The main idea behind fmf was to keep metadata with code so stating 'sources' becomes both obvious and cryptic.

True however it's called based on the 'sources' file name present next to the specfile. If you have a better name please say so.

[thrix]

I would more like syntax:

discover:
  how: fmf
  from: sources

[lukaszachy]

Update: #742 has new code.