Newer createrepo_c doesn't generate comps readable EL7

[kwizart]

I'm using an (older) fedora with koji to build from el7 to fedora rawhide (as the rpmfusion.org project)

Using a backported createrepo_c 1.0.2-2 , the command generate a repomd.xml with a compressed comps.xml, but this file cannot be read by yum that seems to expect an uncompressed comps.xml

/usr/bin/createrepo_c --general-compress-type=gz -vd -o /var/tmp/koji/tasks/8390/618390/repo -i /mnt/koji/repos/el7-free-build/80086/x86_64/pkglist -g /mnt/koji/repos/el7-free-build/80086/groups/comps.xml --update --skip-stat /mnt/koji/repos/el7-free-build/80086/x86_64

Even using the compatibility mode (using createrepo or createrepo_c --compatibility) doesn't change this behavior.

When the koji builder populates the buildroot (for the initial buildfromSCM buildroot), I'm experiencing the following error:

DEBUG util.py:443:  There is no installed groups file.
DEBUG util.py:443:  Maybe run: yum groups mark convert (see man yum)
DEBUG util.py:443:  Failed to add groups file for repository: build - comps file is empty/damaged
DEBUG util.py:443:  Warning: group srpm-build does not exist.
DEBUG util.py:443:  Maybe run: yum groups mark install (see man yum)
DEBUG util.py:443:  Error: No packages in any requested group available to install or update

My current workaround is to manually uncompress the comps file and modify the repodata.xml as appropriate.
After this modification, the build can occurs normally.

[kwizart]

Side note, I had to use --general-compress-type=gz (but I plan to use =xz) because I expect zstd will not work on EL7.
This is patched kojid, but I expect a sane default should be picked as koji is unlikely to have a per build target compression format...

[dralley]

Using a backported createrepo_c 1.0.2-2

But why? Is there any reason to use such a new version of createrepo_c with a distro that will be out of support in 6 months?

[kwizart]

I'm sorry, I think you miss-understood, I'm not using createrepo_c {on} el7, I'm using to generate repodata {for} el7!

[sergiomb2]

libcomps was updated recently (one month ago) , can be related ?

[kwizart]

libcomps was updated recently (one month ago) , can be related ?

no, only updating newer creatererepo_c changes this behavior with no uncompressed comps.xml

Unless we want to deal with koji adapting createrepo_c tasks for each build target, I would suggest the following as createrepo_c is concerned.

• Revert to uncompressed comps.xml for el7 compatibility unless tuned-off
• Revert to bzip2 compressed defaults (or xz that also seems to work with latest RHEL7.x).
• Once EL7 goes EOL by June 2024, revisit theses defaults. (and tune them back if using --compatibility mode).
• Then there is a need to understand why createrepo_c 1.0.0 consumes more memory than the previous versions.

[kontura]

Indeed this is not related to libcomps. These are changes in createrepo_c, some more details are here.

If we introduced the compatibility mode which restores the old defaults would that be sufficient for koji?
It could be used in all build targets.

[dralley]

My current workaround is to manually uncompress the comps file and modify the repodata.xml as appropriate.

Is that workaround not sufficient to last 6 months? Are you editing repodata.xml manually or are you able to use modifyrepo?

Revert to uncompressed comps.xml for el7 compatibility unless tuned-off

I'm fine with adding to the --compatibility flag if its absolutely required.

Revert to bzip2 compressed defaults (or xz that also seems to work with latest RHEL7.x).

On this topic you are mistaken, yum and dnf only looks for the group and group_gz tags. With the previous behavior of createrepo_c, only setting the compression type to gz would create a useful compressed comps file, because the clients were not looking for group_xz (etc). So for any other compression type, yum would download the uncompressed group file. Therefore it's equivalent in effect to disabling compression altogether for comps metadata.

See https://bugzilla.redhat.com/show_bug.cgi?id=1904360 and https://bugzilla.redhat.com/show_bug.cgi?id=2056318

DNF unlike Yum supports the group metadata being compressed (just like any other metadata file, no special behavior, no need to look for alternate tags). But yum is obviously not able to handle group being compressed.

[kwizart]

Is that workaround not sufficient to last 6 months? Are you editing repodata.xml manually or are you able to use modifyrepo?

Looks like a tremendous hack, that would be also needed by the fedora infra, so totally un-appropriate as a workaround.

The kojid usage of createrepo_c(/mergerepos_c) assumes a recent builder to generate content that can be read by all buildroot. (so RHEL7 -> RHEL N and non-EOL Fedora) If this assumption is broken, there is a need to accommodate the tool to build sane defaults for kojid usage.
In kojid context, we cannot assume createrepo_c from the buildroot to be used to generate the appropriate content for the target since we have a chickend and egg problem (createrepo_c comes from the builder userspace).

On this topic you are mistaken, yum and dnf only looks for the group and group_gz tags.

Right, but zck cannot be uncompressed by RHEL7 's yum as I expect.
Then there is the comps issue there, that you have perfectly described.

See also the upstream ticket for koji : https://pagure.io/koji/issue/3945

[kwizart]

Also worth to mention my report to the fedora infra https://pagure.io/fedora-infrastructure/issue/11637
And a(slightly) related ticket for the effects of the option on rawhide users https://pagure.io/releng/issue/11664

[dralley]

Ok, sounds like we want to overload --compatibility then.

Should --compatibility imply --general-compress-type=gz --compress-type=gz also?

[ppisar]

--compatibility is documented as "compatibility with classical createrepo". So whatever createrepo did, createrepo_c --compatibility should do too.

I also cannot see in the manual how to keep an uncompressed format. One may want to have or not to have both compressed and uncompressed formats in the output. Or one may want to have multiple compressions. Current manual reads:

    --compress-type COMPRESSION_TYPE
       Which compression type to use. Supported compressions are: bzip2, gzip, zck, zstd, xz.

   --general-compress-type COMPRESSION_TYPE
       Which compression type to use (even for primary, filelists and other xml).

Now --general-compress-type gz --general-compress-type bz2 produces bzip2 only.

Also the manual is wrong because neither bzip2 or gzip as quoted there are recognized. And zck seems rejected all the time. What's the relation to --zck option?

I personally don't like compressing by default and changing the default compression whenever a new compression type is invented. It breaks compatibility as reported in this issue.

Would you like this interface?:

    --compress-type COMPRESSION_TYPE
       Which compression type to use. Supported compressions are: none, bz2, gz, zck, zstd, xz.
       "none" means no compression. You can use this option multiple times to output in multiple compress types.
       If this option is not used, "--compress-type none" is implied".

[dralley]

Also the manual is wrong because neither bzip2 or gzip as quoted there are recognized. And zck seems rejected all the time. What's the relation to --zck option?

Yes I had also noticed this recently. I'll submit a PR to fix that.

Would you like this interface?:

Are you saying now or in the future? Because currently the combination of --compress-type and --general-compress-type is extremely confusing. #363

[ppisar]

I wrote my comment mainly to deter people from thinking on --compatibility as a moving target which changes compression formats over time as what Fedora considers compatible. It's not fair to other RPM distributions. And to warn from adding new and new options without thinking on a long-term design.

I did not know about #363. My comment would rather fit there. I will copy it there. I also find --compress-type and --general-compress-type confusing.

I'd like to keep this ticket for fixing --compatibility semantics to include gz compression.
However, I'm not against to a large overhaul per #363. Even now. The future can be now.

[dralley]

I wrote my comment mainly to deter people from thinking on --compatibility as a moving target which changes compression formats over time as what Fedora considers compatible. It's not fair to other RPM distributions. And to warn from adding new and new options without thinking on a long-term design.

My understanding is that SUSE has actually supported zstd compressed metadata since SUSE 15, which released before RHEL 8 did. I don't know about Maegia or other distros though.

[Hextremist]

This is a problem for SUSE enterprise 12 SP5 too. This OS is still has long term support from SUSE.

[dralley]

@kwizart @Hextremist Is this sufficient? #418

[kwizart]

I might schedule a test, but I wonder if createrepo will default to --compatibility and produce uncompressed groups and gz for other content. That will permit koji to revert back to createrepo calls that works with older distro (el7) even if using a createrepo_c implementation. (that will avoid a koji patch at all).

[kontura]

I might schedule a test, but I wonder if createrepo will default to --compatibility and produce uncompressed groups and gz for other content. That will permit koji to revert back to createrepo calls that works with older distro (el7) even if using a createrepo_c implementation. (that will avoid a koji patch at all).

The --compatibility flag is not on by default. However it should be possible to set it for all targets so repodata generated with it should be readable everywhere.

[kwizart]

Isn't that the purpose of the createrepo command over createrepo_c to enable the --compatibily flag by default ?
If not, what is the point to keep the createrepo command at all if it does not behave like createrepo ?

From koji perspective, if there is a need to enable the compatibility flag everywhere, there is no point to use creatrepo_c at all but use createrepo instead.

[dralley]

@kwizart If you need that, you'd need to file a PR to change the aliases from a mere symlink to a script

https://github.com/rpm-software-management/createrepo_c/blob/master/createrepo_c.spec#L159-L163

But I don't really understand why this is a problem, only one line needs to be tweaked https://github.com/koji-project/koji/blob/1310f22b8089a011fd4968ef5f7c72cfb8d2f8c3/builder/kojid#L5671 and it's already set up to specify an extra flag there on the createrepo_c side anyway.

And Koji will only need to care about EL7 for a few more months

[kwizart]

I tough it would work using the name of the binary. (like argv[0] == createrepo would enable the --compatibility flag automatically)

It is a problem because koji is a higher level tool and doesn't yet permit using createrepo_c arbitrary parameters. But it has already an option to use createrepo_c or createrepo (same with mergerepo) with the expected parameters.

Using argv[0] worth it because it would allow to use unmodified koji right now.
And defer to later for a better fine grain control to createrepo_c parameters.

Also EL7 lifetime is one thing, but there is also the RHEL extended support and some cases might still use createrepo on newer host to compute repository targeting EL7 for wider community.

[kwizart]

Side note about this issue and mergerepo_c over mergerepo
I wonder if some compatibility flags could be also needed there...

(I will open a separate issue, if needed).

At least there is a good chance that the original issue is fixed. So thanks for that.

[Conan-Kudo]

From the upstream perspective, EL7 is dead to us. The extent of which we should care about it for this project is making it possible to set flags to generate repodata compatible for it, just like we did for EL6.

[Conan-Kudo]

Insofar as issues with Koji, it's worth filing issues there to make it more configurable. But also, please don't run stuff on EL7 anymore, since it is EOL in five months.

[dralley]

Also EL7 lifetime is one thing, but there is also the RHEL extended support and some cases might still use createrepo on newer host to compute repository targeting EL7 for wider community.

RHEL doesn't. Don't know about the wider community e.g. SUSE.

But also, please don't run stuff on EL7 anymore, since it is EOL in five months.

The complaint is more about infra running on new stuff (Fedora) not being able to generate metadata for EL7 easily. Which, prior to this PR, was painful (involved modifyrepo_c steps) and after this PR is still not 100% transparent (requires passing --compatibility)

@kwizart Is it that much more difficult to update Koji? You could probably even reuse use_createrepo_c=False, as legacy createrepo isn't really used anymore.

[kwizart]

@kwizart Is it that much more difficult to update Koji? You could probably even reuse use_createrepo_c=False, as legacy createrepo isn't really used anymore.

That's the very much point of my PR (use_createrepo_c=False) in our (fedora) koji builders. So in one end we can consume zstd compressed metadata, but also produce metadata that can be consumed by yum. Using use_createrepo_c=False and my PR will do the right things easily until we stop caring about EL7 (not before June this year).

The main concern beyond my community project (rpmfusion.org) and fedora Red Hat SuSe or else, is also the wider community (I mean people and entity that create internal corporate build using koji or else for their own use).
Theirs use case will not end in June 2024.

[kwizart]

Once kojid is modified, we can use any per-build / per-tag arbitrary commands to createrepo_c and even drop support for createrepo command there.
Note that the kojid code search for mergerepo_c before mergerepo ,this project cannot revert back to createrepo and mergerepo usage anymore... IMO (it will break assumption in other code).