-
Notifications
You must be signed in to change notification settings - Fork 70
PullApproveAlternatives
Kata Containers uses PullApprove extensively for CI checking its PRs. Unfortunately PullApprove is changing its pricing structure, which will we believe disallow the current Kata free use of PullApprove as a 'free project'.
To that end, let's document what features we currently use on PullApprove, and investigate what alternative options are available.
Note: We currently use PullApprove Version 2. They are now on Version 3, which adds a few features and changes the workflow slightly by binding more tightly to the github review process.
We use the PullApprove signed-off-by option
to check that all commits are signed off, and the signoff email matches the author.
We use this feature heavily to ensure all code contributions conform with the Developer Certificate of Origin, as
detailed in our Contribution docs.
We use the PullApprove always pending feature to check for
WIP or RFC in the PR titles, and/or do-not-merge, wip or rfc labels on PRs, and use PullApprove to block
merging of the PR if any of those is present.
This is a very useful feature for us to easily temporarily prevent accidental merges of PRs that are in flight with known
issues or rework.
Note: the two routes in play here - blocking both via keywords in the PR subject lines, and via labels. Ideally we would continue to support both.
We make use of PullApproves group approval method to define and check we have enough 'acks' from the right number of groups of people to merge a PR.
One feature we use that may not be available in other tools is the ability to have multiple groups, and require acks from all the groups. For instance, we have used this heavily in the past to ensure the documentation team must ack any PRs that make changes to document files.
Being able to count the number of acks from a known trusted group of developers (ideally dictated by a github org group/team membership as can be done with pullapprove) is fairly critical to maintain our merged code quality.
There are a number of different potential routes to examine that may be able to replace the PullApprove functionality:
Github can be configured to check for a required number of reviews. The major lacking feature though is, afaict, lack of support for checking multiple groups, or easily subsetting the members of a project to those who can ack.
We may be able to get a similar effect to multiple group acks by using/supplementing with the github codeowner feature. Elaborated at https://blog.github.com/2017-07-06-introducing-code-owners/
Note also, the github require signed commits feature is not the same as an SoB/DCO signed check. It is a check for crytographically signed commits.
Our own existing checkcommits tool could
be used to supplement some features - such as it already supports the SoB check (but it is normally disabled by default).
checkcommits probably cannot provide all the functionality though, as it ties into part of our CI runtime, and does not
report directly back into github as an 'app', thus it nominally functions at a 'different level'. We should keep it in
mind though in case it is the best/easiest way to supplement another tool to achieve complete functionality for the
project.
Mergify looks like a direct competitor to PullApprove, and is still currently free to open source project. But, what is to say that will be the way forever (as per PullApprove changes in terms).
Probot is a project up on github that helps enable github plugins for robot tasks. It looks like probot may already contain a number of features we would like to use:
- DCO checking
- WIP checking
- mergable configurable plugin looks quite like pullapprove...
Zuul can be configured to assess the required conditions using lightweight jobs and report back results. We should be able to add lightweight (nodeless) jobs to cover all our needs:
- There is a standard job for DCO checking
- Still checking how to do WIP likely using the github label/unlabelled event triggers
See Zuul GitHub driver docs for more details.
Github actions (https://developer.github.com/actions/) (beta)
GitHub Actions uses GitHub API to interact with a repository, some basic validations could be added if this support create blocking PR jobs.
Note: that you can probably run the probot apps as github actions, which may allow us to use 'mergeable' as an action. https://probot.github.io/docs/deployment/#github-actions
| Identifier | Means |
|---|---|
| Y | Yes, supported |
| D | Can be done but requires minimal development |
| N | No, not supported |
| P | Partially supported |
| ? | Unknown |
| What | SoB | WIP | acks | grouped acks | Notes |
|---|---|---|---|---|---|
| PullApprove | Y | Y | Y | Y | third-party service |
| github | N | N | Y | N | already a dependency |
| checkcommits | Y | N | N | N | part of CI, not a GitHub check |
| mergify.io | ? | ? | ? | ? | third-party service |
| probot | Y | Y (P?) | ? | ? | requires hosting the service |
| Zuul | Y | D? | D? | D? | can reuse OSF's Zuul instance |