Your GitHub App to make Buildbot a part of your Pull Request workflow.

Table of Contents

• 1. Scenarios
  • 1.1. Comment /buildbot on Pull Request
    • 1.1.1. Build Log Comment
    • 1.1.2. Check run
    • 1.1.3. Video walkthrough
    • 1.1.4. UML sequence diagram
  • 1.2. Testing
    • 1.2.1. Testing GitHub interaction
    • 1.2.2. TODOs
• 2. Developer Setup
• 3. Useful links
• 4. LLVM links
• 5. Github App documents
• 6. Interacting with Github
• 7. Golang libraries
• 8. Buildbot links
• 9. Misc links
• 10. Lists
• 11. List of figures
• 12. List of tables
• 13. List of code snippets
• 14. TODO
• 15. Terminology

❗	THIS REPOSITORY IS NO LONGER ACTIVELY MAINTAINED.

Let’s build a GitHub App that lets you control your buildbot through /buildbot comments on GitHub Pull Requests.

1. Scenarios

This section lists the ideas and sometimes even fully implemented scenarios we have for this GitHub app.

1.1. Comment /buildbot on Pull Request

In this scenario a user authors a Pull Request comment with the comment body being /buildbot.

The buildbot-app gets notified about a new comment and checks if it matches a regular expression. This EBNF diagram shows the current command syntax:

For the purpose of this demonstration, /buildbot is simply enough.

Internally a command comment will be converted into this structure:

Command structure (cmd/buildbot-app/command/command.go)

// A Command represents all information about a /buildbot command
type Command struct {
    // When true, the command has to pass for the PR in order pass gating
    // (default: true).
    IsMandatory bool
    // Case-sensitive, sorted list of builders without duplicates to run build on.
    // TODO(kwk): Maybe we can default to something reasonable here?
    BuilderNames []string
    // The user's GitHub login that issued the /buildbot comment
    CommentAuthor string
    // When true, we'll try to run the build even if the PR has already been
    // tested at this stage (default: false).
    Force bool
}

There’s a regular expression that a string comment must match (see StringIsCommand()) in order for it to be a valid command string:

Regular Rexpression (cmd/buildbot-app/command/command.go)

const (
    // BuildbotCommand is the command that triggers the buildbot workflow in a
    // GitHub comment.
    BuildbotCommand = "/buildbot"

    // CommandOptionMandatory is the boolean option to make a check run
    // mandatory.
    CommandOptionMandatory = "mandatory"

    // CommandOptionBuilder is the option that can be used multiple times in a
    // command comment. The resulting builders will be a case-sensitive, sorted
    // list of builders with no duplicates.
    CommandOptionBuilder = "builder"

    // CommandOptionForce is the boolean option to enforce a new build even if
    // one is already present.
    CommandOptionForce = "force"
)

// StringIsCommand returns true if the given string is a valid /buildbot command.
func StringIsCommand(s string) bool {
    // force=yes|no : Can be used to allow for PRs to be build even when
    // they are closed or when a check run for the exact same SHA has been
    // run already.
    return regexp.MustCompile(buildRegexPattern()).MatchString(s)
}

// buildRegexPattern returns the regex pattern to match a string against a
// /buildbot command
func buildRegexPattern() string {
    tfOptions := `(yes|no|true|false|f|t|y|n|0|1)`
    mandatoryOption := fmt.Sprintf(`%s=%s`, CommandOptionMandatory, tfOptions)
    forceOption := fmt.Sprintf(`%s=%s`, CommandOptionForce, tfOptions)
    builderOption := fmt.Sprintf(`%s=(\w+)`, CommandOptionBuilder)
    return fmt.Sprintf(`^%s(\s+|%s|%s|%s)*$`, BuildbotCommand, mandatoryOption, forceOption, builderOption)
}

1.1.1. Build Log Comment

The buildbot-app then creates a Thank-you-comment that serves two purposes:

1. It shows the user that we understood the request and are thankful for it and that we are working on it.

2. It is a perfect placeholder to store short build state changes for future lookups. That is why we call this comment the build-log-comment.

   

   Just imagine, your PR gets updated and you want to see the previous build results. The build-log-comment is there for you too look it up.

The code for creating the comment is straight-forward:

Thank You! (cmd/buildbot-app/on_issue_comment_event.go)

     // This comment will be used all over the place
        thankYouComment := fmt.Sprintf(
            `Thank you @%s for using the <a href="todo:link-to-documentation-here"><code>%s</code></a> command <a href="%s">here</a>! `,
            command.BuildbotCommand,
            *event.Comment.User.Login,
            *event.Comment.HTMLURL,
        )

        newComment, _, err := gh.Issues.CreateComment(context.Background(), repoOwner, repoName, prNumber, &github.IssueComment{
            Body: github.String(thankYouComment +
                `<sub>This very comment will be used to continously log build state changes for your request. We decided to do this in addition to using Github's Check Runs below so you can inspect previous check runs better.</sub>`,
            ),
        })

1.1.2. Check run

Of course, we are also using GitHub’s check runs as you can see here:

ℹ️	I really like that we can dynamically create check runs on request and give them good names.

When you click on Details next to a check run, you’re brought to this page on GitHub:

1.1.3. Video walkthrough

We walk you through the creation of a Pull Request and authoring the /buildbot comment in this in this short video: https://www.youtube.com/watch?v=9NpbKEmkvt8

1.1.4. UML sequence diagram

The sequence diagram for this scenario is layed out here. It includes some of the internals of the processing.

1.2. Testing

1.2.1. Testing GitHub interaction

We’re using a fantastic library to run to simulate sequential GitHub interaction: https://github.com/migueleliasweb/go-github-mock.

For example, when /buildbot comment is authored on a pull request we don’t want a build to run if the pull request is not mergable. Therefore we first have to take the event input and get the pull request from GitHub before we check if is mergable:

Get PR and check mergability (cmd/buildbot-app/on_issue_comment_event.go)

     commentUser := *event.Comment.User.Login
        repoOwner := *event.Repo.Owner.Login
        repoName := *event.Repo.Name
        prNumber := *event.Issue.Number
        pr, _, err := gh.PullRequests.Get(context.Background(), repoOwner, repoName, prNumber)
        if !pr.GetMergeable() {
        }

In order to test that a PR is not mergable, we can simply create a valid github.PullRequest object (see prOK()) and set the Mergable member to false. The mock server will return it as the first request and afterwards create a POST a comment about the pull request not being mergable:

Test: Get PR and check mergability (cmd/buildbot-app/on_issue_comment_event_test.go)

func TestOnIssueCommentEventAny(t *testing.T) {
    t.Run("pr not mergable", func(t *testing.T) {
        t.Run("comment writable", func(t *testing.T) {
            prNotMergable := prOK()
            prNotMergable.Mergeable = github.Bool(false)
            srv := NewMockServer(
                // Get PR for comment event
                mock.WithRequestMatch(
                    mock.GetReposPullsByOwnerByRepoByPullNumber,
                    prNotMergable,
                ),
                // Create comment on about PR not being mergable
                mock.WithRequestMatch(
                    mock.PostReposIssuesCommentsByOwnerByRepoByIssueNumber,
                    github.IssueComment{
                        Body: github.String("blabla"),
                    },
                ),
            )
            fn := OnIssueCommentEventAny(srv)
            err := fn("1234", "created", issueCommentEventOK())
            require.ErrorContains(t, err, "pr is not mergable", "expected and error because pr is not mergable, yet")
        })
    })
}

For this trick to work we have to use dependency injection by passing a Go interface (Server) instead of a real server object to functions in various places:

Server interface (cmd/buildbot-app/server.go)

// Server specifies the interface that we need to implement from the AppServer
// object in order to provide a decent mock in tests.
type Server interface {

    // NewGithubClient returns a new GitHub client object for the given
    // application ID.
    NewGithubClient(appInstallationID int64) (*github.Client, error)

    // RunTryBot runs a "buildbot try" command
    RunTryBot(responsibleGithubLogin string, githubRepoOwner string, githubRepoName string, properties ...string) (string, error)
}

1.2.2. TODOs

• ❏ Reset check run to neutral after Pull Request was updated.

• ❏ Deal with buttons shown at the top of check run details page.

2. Developer Setup

I’m using a Fedora Linux 37 on my local machine and for most of the containers.

$ git clone https://github.com/kwk/buildbot-app.git && cd buildbot-app
$ sudo dnf install -y direnv golang podman podman-compose buildbot pandoc asciidoctor
$ gem install asciidoctor-lists pygments.rb
$ go install github.com/cespare/reflex@latest
$ cat <<EOF >> ~/.bashrc
export PATH=\${PATH}:~/go/bin
eval "\$(direnv hook bash)"
EOF
$ source ~/.bashrc
$ direnv allow .
$ make infra-start
$ make app

• Clone the repo.

• Install tools we need/use for development locally. If this was a deployment site the only requirement is buildbot so that the github app can make a call to buildbot try.

• Install extension to create list of figures etc. and install pygments for source code highlighting.

• Install hot-reload tool.

• Make tools above available upon next source of .bashrc.

• Reload .bashrc to have direnv and reflex working in your current shell.

• Navgigate out and back into the project directory to have direnv kickin. If this doesn’t work, try direnv allow ..

• Bring up local containers for a buildbot setup with one master and three workers.

• Run and hot reload the app code upon changes being made to any of your *.go files or your .envrc file.

3. Useful links

4. LLVM links

• Discussion on LLVM Discourse: https://discourse.llvm.org/t/rfc-prototyping-pre-commit-testing-using-buildbot/69900?u=kwk

5. Github App documents

• Github Webhook Events and Payloads: https://docs.github.com/en/webhooks-and-events/webhooks/webhook-events-and-payloads

• Github Apps documentation: https://docs.github.com/en/apps

6. Interacting with Github

• Forwarding Github Webhooks to your local dev machine: https://dashboard.ngrok.com/get-started/setup

• Github Emoji Cheat Sheet: https://github.com/ikatyang/emoji-cheat-sheet/blob/master/README.md

7. Golang libraries

• For using Github API v3 from Golang: https://github.com/google/go-github

• GraphQL Go Library for Github API v4: https://github.com/shurcooL/githubv4

• For mocking the above repo responses: https://github.com/migueleliasweb/go-github-mock

• Go web framework: https://github.com/labstack/echo

• For handling github events: https://github.com/cbrgm/githubevents

• For authentication of Github App from private key file: https://github.com/bradleyfalzon/ghinstallation

8. Buildbot links

• System Architecture: https://docs.buildbot.net/latest/manual/introduction.html#system-architecture

• Custom services (Might be worth looking into): https://docs.buildbot.net/latest/manual/configuration/services/index.html

9. Misc links

• Recording terminal sessions: https://github.com/faressoft/terminalizer

• For automatic reloading: https://github.com/cespare/reflex

• Per-Directory environment files: https://direnv.net/

10. Lists

11. List of figures

12. List of tables

13. List of code snippets

Listing 1. Command structure (cmd/buildbot-app/command/command.go)
Listing 2. Regular Rexpression (cmd/buildbot-app/command/command.go)
Listing 3. Thank You! (cmd/buildbot-app/on_issue_comment_event.go)
Listing 4. Get PR and check mergability (cmd/buildbot-app/on_issue_comment_event.go)
Listing 5. Test: Get PR and check mergability (cmd/buildbot-app/on_issue_comment_event_test.go)
Listing 6. Server interface (cmd/buildbot-app/server.go)

14. TODO

• ❏ properly document developer setup with ngrok and how to setup the .envrc file

• ❏ hook into buildbots event system and send feedback to buildbot app from there?

15. Terminology

PR or Pull Request

"Pull requests let you tell others about changes you’ve pushed to a branch in a repository on GitHub. Once a pull request is opened, you can discuss and review the potential changes with collaborators and add follow-up commits before your changes are merged into the base branch."  — (About pull requests)

Buildmaster or Buildbot Master

"Buildbot consists of a single buildmaster and one or more workers that connect to the master. The buildmaster makes all decisions about what, when, and how to build." — (Buildbot System Architecture)

Buildbot Worker

"The workers only connect to master and execute whatever commands they are instructed to execute." — (Buildbot System Architecture)

Builder

"A builder is a user-configurable description of how to perform a build. It defines what steps a new build will have, what workers it may run on and a couple of other properties. A builder takes a build request which specifies the intention to create a build for specific versions of code and produces a build which is a concrete description of a build including a list of steps to perform, the worker this needs to be performed on and so on." — (Buildbot System Architecture)

Scheduler

"A scheduler is a user-configurable component that decides when to start a build. The decision could be based on time, on new code being committed or on similar events." — (Buildbot System Architecture)

Reporters

Reporters are user-configurable components that send information about started or completed builds to external sources. Buildbot provides its own web application to observe this data, so reporters are optional. However they can be used to provide up to date build status on platforms such as GitHub or sending emails. — (Introduction)