Add warning status for checks

[rousseldenis]

Referring to this issue: isaacs/github#1465

Should be a great feature!

Thanks

UPDATE: This is the original issue description:

When creating a status on a commit, there are only four choices: error, failure, pending, or success.

https://developer.github.com/v3/repos/statuses/

It would be useful to have a level called "warning". If a commit has statuses that are all success or warning, then an orange exclamation mark should be used to annotate the commit, instead of a red X. If there is at least one failure, then you should get the red X.

This would clean up the implementation of "recommended but not required" checks for PRs. For example, in unicode-org/icu, we have a recommended policy that a PR contains only one commit. We currently have this set up as a non-required status that fails, which causes the whole PR to get the red X, even when all of the other checks are green.

[xmo-odoo]

@rousseldenis you should copy over the entire description, possibly reworded or augmented with feedback from the comments, because it's uncomfortable to have to go over the linked issue every time in order to see / remember what the subject and justification actually is.

And since the isaacs repo has been archived it's not even possible to react (let alone comment), so the link really has no value other than archival.

[rousseldenis]

@xmo-odoo I've updated the description

[mattyb]

One use case for this would be visual diffs, where a detected difference requires review but doesn't necessarily mean there's an error.

[kuteninja]

I would use this also on our terraform repository, on cases where terraform plan works correctly and at least one element is going to be destroyed by it when applied. Just as an extra precaution.

[mturley]

This would also be useful for unit test coverage checks, where we don't want to require a coverage percentage to merge but we still want to be alerted to low coverage.

[rjeffman]

It is also useful when testing for future versions to be supported (e.g.: Python 3.12 as of today). CI should not break if a beta version fails, but should warn that some attention is needed.

[rousseldenis]

Any update on this ?

[Quba1]

Is there any new information about this feature?

[iliya-malecki]

any updates on this?

[mattcobb]

The general case for this would be any check that fails, but isn't set to block merging. You could drop the required to merge wording. Anyone visual can't see passed the red to read that anyway. I've worked with devs that been looking at PRs for years with red Xs and didn't realize the check was merge optional.

[daltonv]

This would help enable a setup where only CI blocks code that generates warnings from merge. Something like talked about here https://embeddedartistry.com/blog/2017/05/22/werror-is-not-your-friend/

[phyzical]

Our usecase is around implementing new checks for example say we implement a CVE checker, but don't want to make it blocking out the gate. we currently have to mark it as green which results in people forgetting to actually action them to a green state to start enforcing the checks.

I feel if it was always "warning" (orange) people would atleast notice it every now and then providing a nudge

[Julienraptor01]

This is smth that would really be nice
i would say it's actually even needed, because otherwise some people don't see those

[h-mayorquin]

It would be great to have this.

[tanmayshishodia]

any update on this issue?

[rousseldenis]

In fact, we have solved this in https://github.com/OCA repositories by using a separate check. We used a separate tool (https://github.com/acsone/checklog-odoo) to parse logs for warnings and errors. I think this workaround can be used in most of projects. My two cents.

[jhdark]

Any update on this? We want to test our code in a CI with the latest versions of certain version specified dependencies, to give future warning of which features may no longer work, but now every commit in a PR fails because of this. Would be good to just have it pass but with warning

[ljharb]

This will also be useful for cron jobs that succeed, but may or may not need to "do something" based on the results. In other words, I'd love to see a green checkmark when the followup action is completed, and a warning indicator when the task succeeded but nothing needed to be done. It'd make it trivial to visually scan a list of runs and see which ones actually did something, which ones failed, and to ignore the ones that succeeded but didn't do anything.

[eraliod]

This would also be used in our terraform repo. Our plan step can yield warnings. In that case, the plan gets a green check mark, but the apply step would fail. Warnings must be dealt with before PR is approved.
The reason we configure warnings is that terraform would stop at the first error, and we wouldn't get a full set of issues.

[infogulch]

Could you add a step after terraform runs to check if it output warnings and create an error if it did?

[eraliod]

Thanks for the callout. I do that already like so...

minimum code here. But I put this in a step after my plan. So I can still go back and "review" the human readable plan if this thing fails

if terraform plan -refresh=false -json | jq -e -r 'select(.["@level"] == "warn")' > /dev/null 2>&1; then
echo "warning found"
exit 1
fi

But I still would rather have a status in github checks that indicates "passed with warnings"