Add some automation for doing periodic mass rebuilds of Fedora packages #864
Conversation
These mass rebuilds will help test the snapshot builds and catch regressions in real-world projects. The GitHub workflows are currently configured to start a new mass rebuild once per month and then when the rebuild is complete, it will create any issue reporting any regressions since the last build. The goal is to keep the rebuilder as simple as possible, so it will only rebuild packages that successfully built in the last rebuild, and it makes no attempt to work around packages that hard code gcc as their compiler. As a result, some of the 'passing' builds may actually use gcc instead of clang, but this is an acceptable trade-off to keep the rebuild process simple.
.github/workflows/rebuilder.py
Outdated
| # latest is a bit a successful build, but this doesn't mean it failed. | ||
| # It could be in progress. |
There was a problem hiding this comment.
This comment looks unfinished.
.github/workflows/rebuilder.py
Outdated
| return project_name | ||
| except: | ||
| continue | ||
| print("FAIL") |
There was a problem hiding this comment.
I suggest to use import logging and logging.warn("Fail") instead. This makes debugging the logs much easier. Also, please log more things so we know better where the script currently is.
There was a problem hiding this comment.
@tstellar I know that the CI system is buggy at the moment but I encourage you to add tests to your script. At least some doc tests which should be carried out as regular tests in our setup. I know this is hard for functions that involve copr and alike.
| owner: context.repo.owner, | ||
| repo: context.repo.repo, | ||
| title: "Mass Rebuild Report", | ||
| labels: ['mass-rebuild'], |
There was a problem hiding this comment.
Please create mass-rebuild label first.
There was a problem hiding this comment.
Done: mass-rebuild
Co-authored-by: Konrad Kleine <[email protected]>
Co-authored-by: Konrad Kleine <[email protected]>
| if p["name"] not in pkgs: | ||
| continue | ||
| if not latest_succeeded: | ||
| pkgs.discard(p["name"]) |
There was a problem hiding this comment.
I wonder if we should have a small list of important packages that should never get removed even if they failed last time, e.g. qemu.
There was a problem hiding this comment.
I was planning to add tmt tests for important packages, like qemu, so that they get tested every day, not just periodically with the mass rebuild, but I agree, I think it does make sense to ensure they always get tested by this script too.
These mass rebuilds will help test the snapshot builds and catch regressions in real-world projects. The GitHub workflows are currently configured to start a new mass rebuild once per month and then when the rebuild is complete, it will create any issue reporting any regressions since the last build.
The goal is to keep the rebuilder as simple as possible, so it will only rebuild packages that successfully built in the last rebuild, and it makes no attempt to work around packages that hard code gcc as their compiler. As a result, some of the 'passing' builds may actually use gcc instead of clang, but this is an acceptable trade-off to keep the rebuild process simple.