Add variant support

[travier]

cmd-init: Remove support for subdirectory

Remove that support as:

• we currently don't use this feature,
• we chose another path for multiple version support in openshift/os as
  using subdirectories lead to a lot of additionnal complexity.

See: openshift/os#803

This reverts commit 4781e46.

---

cmd-*: Support config repos with multiple variants

On top of the defaut variant, a config repo may include aditional
optional variants.

The variant used is selected at cosa init with the --variant option
and the value is recorded locally in src/config.json.

See:

• Split manifest and update CI openshift/os#852
• manifests: Prepare for variant support in COSA and add RHEL 9.0 & SCOS variants openshift/os#901
• Updates for variant & yumrepo support in COSA openshift/os#958

Reworks: 480c239 init: Require specifying a config repository

---

kola: Add osversion (variant) support for denylist

Enable specifying osversions (rhel-8.6, rhel-9.0, c9s) to skip specifc
tests in the kola denylist.

We do not use the varaitn names here as they are not stable accross
versions thus we require the version to be explicit.

---

schema: Add coreos-assembler.config-variant

Make sure that we record which variant was built.

---

schema: make schema & make vendor for mantle

[travier]

Alternative to #1459

[travier]

/retest

[travier]

I can also rework this PR to move the script into COSA and store the RHEL version into a file instead.

[jlebon]

Alternative in #2936.

[travier]

I've reworked the logic here to be declarative instead of relying on a script while keeping some of the properties that we have right now.

This introduces variant_foo config files that are used to store the name of the manifests corresponding to a variant. This enables us to have a default variant (variant_default) pointing to CentOS Stream, an rhcos variant (variant_rhcos) pointing to a given RHEL version and other explicitly versioned variants that point to a given RHEL/CentOS version (rhel-8.6 (variant_rhel-8.6), rhel-9.0 (variant_rhel-9.0), c9s (variant_c9s)).

• If no variant is explicitly selected then the default manifest.yaml will be used. This is to keep the current setup for FCOS.
• If there is no manifest.yaml file provided by default in the repo, then COSA will read the variant_default file to figure out which variant to use by default.
• If you explicitely specify a variant as parameter then COSA will read the corresponding variant_$VARIANT file and use that as default instead, overriding any default set in the repo.

[travier]

Example setup in openshift/os#855

[travier]

Example with SCOS: openshift/os#856

[travier]

The main advantage of doing things this way is that we can have the internal CI always use the rhcos variant and thus never need an update when we change the default version of RHCOS. Changing the default is entirely based on a single symlink in the openshift/os repo.

We can then switch the repo default to be SCOS and community contributors will get the CentOS Stream version by default.

[cgwalters]

We can then switch the repo default to be SCOS and community contributors will get the CentOS Stream version by default.

Hmm. Right, though I'm thinking we're trying to overload two things into one switch:

• distro: centos vs rhel
• major.minor: rhel 8.6 vs 9.0

I'd agree that it could make sense to logically switch the default distro to centos stream. But, we can have pipelines which want to build RHCOS set a flag (something like cosa init --variable=distro=rhel) that overrides that.

I think we want the default major.minor version is better hardcoded in openshift/os, but we also want something like cosa init --variable=major=9 or so so that pipelines can target rhel9.

OTOH, hopefully in a year or so we won't care at all about --variable=major=8 at all, which will help significantly with the matrix here.

(One thing I think we should also keep in mind here is...what if we switched fedora-coreos-config to not use git branches either? The lockfiles and CI there are oriented around it, but clearly this whole approach could work there too. It'd just be good to have the two work the same way. I'm not saying we should scope this in or block on it, but reiterating I think it would be a better end state to have the two work the same way)

[cgwalters]

I definitely want to make progress here and not block on a perfect design...but OTOH the semantics we choose here have a lot of ramifications and will get tied into various pipelines and be something everyone needs to understand, so that motivates some level of design discussion.

OTOH, I think we can change this later too without too much trouble, would just require some "ratcheting" across cosa and the pipelines.

[travier]

The UX with this one would be:

# Get the default for the repo
$ cosa init https:/...../ repo.git
# Get the default for RHCOS
$ cosa init https:/...../ repo.git rhcos
# Get a specific version as explicitly asked
$ cosa init https:/...../ repo.git rhel-8.6
$ cosa init https:/...../ repo.git rhel-9.0
# Get the default for SCOS. Remember that we will switch SCOS to CentOS Stream 10 at some point too
$ cosa init https:/...../ repo.git scos
# Get a specific version as asked
$ cosa init https:/...../ repo.git c9s

All RHCOS pipelines will thus use the rhcos one. All SCOS pipeline will use the scos one. Other test jobs can be more explicit.

[travier]

Note: CI is failing on an unrelated issue and is not testing that change currently.

[jlebon]

This is better than a script, though I don't like that it dirties the config dir. And more generally, that it retains state in the config dir at all. Ideally state would remain in the cosa workdir instead and the config dir remains wholly described by the git SHA like today, which is an important property.

I like the variables idea from @cgwalters. I.e. we still have a single manifest.yaml, but we support overriding default variables there, and the manifest conditionally includes things based on those. We then keep the values in src/vars.yaml alongside src/config.

It needs some more fleshing out though, because there's also image.yaml and extensions.yaml involved which don't support this. Maybe we can have a special variant variable which is used in the treefiles, but also for finding e.g. image-${variant}.yaml?

[jlebon]

Anyway, short-term I'd still personally prefer the subdirs approach. The cosa patch for it is not that large. But I'm OK with this if we really want. Won't mention this again now because I don't want to block this!

[jlebon]

Though looking at this patch, I think we could enhance it slightly to avoid the few symlinks it does by keeping the variant active in e.g. src/variant or something? And then e.g. buildextend-extensions knows to read that file to get the path to extensions.yaml to hand over to rpm-ostree?

[cgwalters]

Ideally state would remain in the cosa workdir instead and the config dir remains wholly described by the git SHA like today, which is an important property.

But after this lands, note the system state won't be fully described by the git sha; imagine that one wants to do cosa buildprep on top of a previous build for e.g. RHCOS. What will happen now is that the user will also need to manually cosa init to the target variant.

I think what we want is to have this information inside meta.json for example, and then cosa buildprep would also use it.

[travier]

We've discussed that today and I'll update this PR to be closer to #1459 and what's in the comments above (recording the variant in the meta.json).

[travier]

Matching RHCOS PR: openshift/os#958

[cgwalters]

Awesome, thanks so much for persevering on this!

LGTM

[travier]

I need to re-do testing with all variants

[jlebon]

Should we support --variant default being equivalent to not passing a variant? This could be useful in scripting where the variant passed to cosa init is a variable.

Not a blocker of course. Good to do in a follow-up.

[jlebon]

LGTM!

[travier]

(Still planning to do a last round of testing but haven't had the time yet).

[travier]

I've tested that with 4 variants and it worked so far. Will merge on Monday to be able to do follow-up changes as needed.