Build Status: Improve automation #148
Comments
|
There is also a gap in automation for existing projects, causing more serious troubles. CI/GHA jobs regularly hibernate, because of the reason outlined below. Important Please make sure to also add this obstacle to your personal recurrent check list. 🙏
|
There is a longer discussion in https://stackoverflow.com/questions/67184368/prevent-scheduled-github-actions-from-becoming-disabled about work-arounds. |
|
GitHub Workflow Immortality sounds nice, thanks! Now, who is going to implement it? ;] /cc @bmunkholm |
Evaluating the outcome automatically, and making us aware of any significant events@kneth suggested the most important detail here would be to automate the evaluation of the outcome from builds of multiple projects. @amotl agrees. Until we have it, he kindly agreed to volunteer to check the Build Status regularly, and act or dispatch accordingly. Idea 1We are suggesting to possibly aggregate the outcome from multiple projects, and to submit a notification to Slack or the crate-alerts repository when there is an event that will need attention, which is probably the right choice. This variant might need additonal thoughts, because you would need to consider flakyness and hysteresis aspects. Idea 2An alternative to the push-notification-on-error would be to just post a summary into a dedicated Slack channel each morning, of all jobs that are failing. That would be much easier, because it's a straight-forward pipeline without needing to define what a significant event is: In both cases, we need to pick a place where to run the automation unattended. In a perfect world, everything would be self-contained within a single GHA workflow definition. |
This part of the pipeline would probably be best implemented using a little Python program, which uses a list of direct URLs to arbitrary GHA workflows files as input, goes through them, invoking a call to the GitHub API to find out about their outcomes, and return a summary, optionally filtered by status? |
GitHub repository names from Build Status MarkdownCurrently, the source of the Build Status page is just Markdown. This little Bash function extracts a list of GitHub repository names. It uses HTTPie and Perl. function build-status-gh-repos() {
http --follow https://github.com/crate/crate-clients-tools/raw/refs/heads/main/docs/status.md | \
grep "github.com.*yml" | perl -pe 's#.*github\.com/(.+?)/(.+?)/.*#\1/\2#' | sort | uniq
}The command generates the list of all relevant public-facing repositories, including artefacts we or others are shipping that work well together with CrateDB, or include relevant integration tests, without crate/crate itself. Demobuild-status-gh-reposGHA workflow status (per-repository)This little function written in Bash acquires relevant information about recently failed scheduled workflow runs for a specific repository from GitHub's API. It uses HTTPie and jq, and does not require a GitHub authentication token for a few requests until it trips a rate limit. function gha-failures() {
repo=$1
http ${HTTPIE_OPTIONS} https://api.github.com/repos/${repo}/actions/runs \
event==schedule status==failure "created==:>=$(date -d 'yesterday' '+%Y-%m-%d')" | \
jq '[ .workflow_runs[] | {name, display_title, path, html_url, status, conclusion, created_at} ]'
}Note The parameter Demogha-failures crate/cratedb-examples[
{
"name": "AutoML",
"display_title": "AutoML",
"path": ".github/workflows/ml-automl.yml",
"html_url": "https://github.com/crate/cratedb-examples/actions/runs/12196847284",
"status": "completed",
"conclusion": "failure",
"created_at": "2024-12-06T10:05:50Z"
}
]Optionally use authentication when running into rate limits. It's an (invalidated) personal access token (classic), using the "workflow" scope. Slack NotificationsA quick search discovered those GitHub <-> Slack integrations. If some of it are capable enough, we may even think about omitting the pre-aggregation/-summarization step above. Those seem to be the canonical ones.
Other people at Crate.io are already using GitHub's scheduled reminders, to converge relevant information into a Slack channel each morning at 9 am. |
All at onceThe program cratedb-ecosystem-build-failures.sh includes both functions presented above, and can be used as the baseline for ad hoc explorations and for conducting orientation flights. Output is still rough, and improving it will probably need a rewrite in pure Python. Installwget https://gist.github.com/amotl/d7de8de01eafba602dd9c6bfa9cfac6d/raw/cratedb-ecosystem-build-failures.shUsageThe GITHUB_TOKEN is an (invalidated) personal access token (classic), using the "workflow" scope. Create your own one per https://github.com/settings/tokens. GITHUB_TOKEN=ghp_00r4G0tGxLDT5RGOwWNw7tZhFnK5fT0uMfoo
HTTPIE_OPTIONS="--auth-type bearer --auth ${GITHUB_TOKEN}"
bash cratedb-ecosystem-build-failures.sh |
|
Just sharing a little walkthrough that uses gh to inquire job runs and their outcomes using the GitHub API from the command line. List failed job runsRecent failures, max. 20gh run list --repo=crate/cratedb-toolkit --event=schedule --status=failure --limit=20Recent failures, since yesterdaygh run list --repo=crate/cratedb-toolkit --event=schedule --status=failure --created='>2025-02-09'Inspect job runDisplay jobsgh run view --repo=crate/cratedb-toolkit 13232774906Display logs of failed workflow stepgh run view --repo=crate/cratedb-toolkit 13232774906 --log-failed |
|
It looks like using |
|
Yeah, About
UsageDownload file enumerating list of curated repositories to inspect. open https://github.com/crate/cratedb-github-summary/raw/refs/heads/main/cratedb-repositories-ecosystem.txtGenerate report about failed CI runs within now-24h. export GITHUB_TOKEN=ghp_600VEZtdzinvalid7K2R86JTiKJAAp1wNwVP
python rapporto.py qa --repositories-file=cratedb-repositories-ecosystem.txtExampleOpen example reportQA report 2025-02-10A report about GitHub Actions workflow runs that failed recently (now-24h). Scheduled
Pull requestsDynamic |
About
This is a proposal to improve the sitation around the Build Status page on different ends.
Q&A
@kneth asked the other day:
Originally posted by @kneth in #146 (review)
The text was updated successfully, but these errors were encountered: