How to apply a section of main.fmf only to plans/tests

[LecrisUT]

I want to make it simple to apply a system specific configuration, e.g.:

provision:
  how: container

Since there is no system/user configuration, one way to do this is to apply it project wide, e.g. putting this into /main.fmf. This works well, except that these are picked up by tests as well and show up as warnings that it is not recognized. Is there a way to write a main.fmf file that has specific sections for plans/tests only?

As a workaround I could add this to a plans specific folder, but this is not general enough that the same method could be applied to any project.

[psss]

Thank you for the question, @LecrisUT. The config example above is relevant for plans only so the most natural place for this common metadata seems to be the /plans/main.fmf file which is automatically inherited to all plans. This is exactly how the fmf inheritance feature is intended to be used.

As a workaround I could add this to a plans specific folder, but this is not general enough that the same method could be applied to any project.

I don't understand the concern of not being general enough. Could you please elaborate a bit more? The inheritance is quite flexible, you can place the shared information which should not be duplicated to the right place in the fmf.Tree. The following examples could give you some inspiration:

• https://tmt.readthedocs.io/en/latest/examples.html#share-common-attributes
• https://tmt.readthedocs.io/en/latest/examples.html#inherit-plans

I'm also writing a new chapter of the tmt Guide which will describe the concept in more detail:

• https://github.com/teemtee/tmt/pull/1987/files#diff-09491514b37aab1b65df8651a1ff9332f0f6af474defbbc2834a489907a2011aR436

Finally, you can get some more inspiration from the real-life examples from the tmt project itself:

• https://github.com/teemtee/tmt/blob/main/plans/main.fmf
• https://github.com/teemtee/tmt/blob/main/plans/features/main.fmf

Hope this helps.

[LecrisUT]

The issue is that you might have a /plans/main.fmf already in vcs, e.g. to manage prepare stage for each plan. But then you want to allow for user configurations, e.g. /main.fmf containing provision information. But then that would propagate to the tests as well. If we can make main.fmf sections apply only to plans.

As a workaround, after some playing around I found that you can use:

/plans:
  provision:
    how: container

For some reason it failed to me when I tried to set enabled: false (I think I did not account for the path correctly) so I thought that did not work. But indeed that works, so this is not as relevant anymore.

---

The only issue here now is how to make it general. E.g. ProjectA has paths:

/plans/rpmlint:
/plans/unit-test:
/tests/testA:

And project B has paths:

/tmt/collection/rpmlint:
/tmt/collection/unit-test:
/tmt/tests/testA:

Now let's consider that we want to have packit automatically setup a local test environment with provision.how: container, etc. The issue is that we cannot create a simple main.fmf file in the root because it will apply for both plans and tests. And we cannot impose a specific format. This is what I mean with using a specific path is not general enough.

[psss]

Could you please explain what exactly do you mean by "user configurations"? Both prepare and provision step configuration are defining how the plan should be executed. It does not depend on where you store this information. It can be in the root main.fmf under a virtual hierarchy using /plan or it can be in /plans/main.fmf.

As for now the provision step config is only used when executing tests from your laptop. When tests are run in Testing Farm using packit the provision.how part is ignored and overriden with provision --how connect. We plan to have enhancement to allow better way to actually respect the provision step config when tests are being run in Testing Farm.

Also, if you refer to user configuration which should be common across multiple projects / git repositories, in #84 there's some initial kick-off of the discussion about the feature. @lukaszachy, @happz, @thrix, any comments here?

[LecrisUT]

Could you please explain what exactly do you mean by "user configurations"?

User configuration here is similar to the ones described in the documentation, however it project specific. I.e. the order preference:

• system global config /etc/tmt (not implemented)
• user global config ~/.config/tmt (not implemented)
• user project config <FMF_Root>/main.fmf
  • already available
  • is not included in VCS (.gitignored)
• vcs project config: everything above
  • maintained via vcs

Normally we would want the user project config and vcs managed ones to have the opposite priority, but that would require a more complex design. For now though, this approach is already capable of doing a lot of the heavy work.

Both prepare and provision step configuration are defining how the plan should be executed. It does not depend on where you store this information. It can be in the root main.fmf under a virtual hierarchy using /plan or it can be in /plans/main.fmf.

The key part is what is under vcs /plans/rpmlint.fmf, /plans/unit-tests.fmf, etc. and what are intentionally left unversioned for the local user to use to run tests locally /main.fmf.

As for now the provision step config is only used when executing tests from your laptop.

This is exactly where we would want the user (project) configuration.

The issue here is how to design a /main.fmf that can be applied to any project and delegate the tmt commands to a cli tool (e.g. packit tests run -> tmt tests run). Wrapping every tmt --flag option is not a good design option, but using a /main.fmf is almost perfect, except for the ambiguity of where the plans are defined.