Skip to content
This repository was archived by the owner on Nov 18, 2021. It is now read-only.

Commit

Permalink
ci: move to build branch model
Browse files Browse the repository at this point in the history
Currently, runtrybot triggers a repository_dispatch build via the
test_dispatch workflow.

This dispatch workflow then starts the process of updating the CL with a
"starting" notification, running the build matrix, and then finally
updating the CL with the result state.

However, because this all happens as part of the dispatch workflow, it
all happens using the _tip_ definitions of the dispatch workflow, not
the definition in the commit under test.

This means that when we need to make changes to the workflow definition,
the try bot result from testing that CL is not the result of using the
changes to the workflow in that commit. Hence we must blindly submit
that CL and hope that it doesn't break the workflow (the next tip build
will tell us but still). This is clearly very brittle.

This change switches us to a model of using build branches that are
created by the initial repository_dispatch. These branches then trigger
a regular branch build, albeit using a special git ref
(ci/$CHANGE_ID/$COMMIT). Status updates to the corresponding CL happen
as before, but this time from the build branch workflow. When a CL build
branch workflow has completed, the corresponding build branch is deleted
(regardless of the test result).  Note that for now we do not delete
this branch: we first want to ensure that a master build succeeds.

Note that from a security perspective this is fine (tm). In order to
trigger a repository_dispatch event in the first place a user must have
write permission to the CUE repo. So there is no privilege escalation
here.

The initial repository dispatch is triggered by a user with write
privileges, but then the subsequent push of the build branch happens as
cueckoo.

Closes #513

Change-Id: I2738fc488d6a8ef08e7e83b151b12934b9f1ee15
Reviewed-on: https://cue-review.googlesource.com/c/cue/+/8212
Reviewed-by: Paul Jolly <paul@myitcv.org.uk>
  • Loading branch information
myitcv committed Jan 15, 2021
1 parent bef7b26 commit ccbfbd2
Show file tree
Hide file tree
Showing 5 changed files with 270 additions and 248 deletions.
5 changes: 4 additions & 1 deletion .github/workflows/rebuild_tip_cuelang_org.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,10 @@ on:
- master
jobs:
push:
runs-on: ubuntu-latest
runs-on: ubuntu-18.04
defaults:
run:
shell: bash
steps:
- name: Rebuild tip.cuelang.org
run: curl -f -X POST -d {} https://api.netlify.com/build_hooks/${{ secrets.CuelangOrgTipRebuildHook
Expand Down
10 changes: 8 additions & 2 deletions .github/workflows/release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,10 @@ on:
- v*
jobs:
goreleaser:
runs-on: ubuntu-latest
runs-on: ubuntu-18.04
defaults:
run:
shell: bash
steps:
- name: Checkout code
uses: actions/checkout@v2
Expand All @@ -21,7 +24,10 @@ jobs:
args: release --rm-dist
docker:
name: docker
runs-on: ubuntu-latest
runs-on: ubuntu-18.04
defaults:
run:
shell: bash
steps:
- name: Check out the repo
uses: actions/checkout@v2
Expand Down
51 changes: 42 additions & 9 deletions .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,12 +4,9 @@ name: Test
on:
push:
branches:
- '*'
- '**'
tags-ignore:
- v*
defaults:
run:
shell: bash
jobs:
test:
strategy:
Expand All @@ -20,11 +17,16 @@ jobs:
- 1.14.9
- 1.15.x
os:
- ubuntu-latest
- macos-latest
- windows-latest
- ubuntu-18.04
- macos-10.15
- windows-2019
runs-on: ${{ matrix.os }}
defaults:
run:
shell: bash
steps:
- name: Write the gitcookies file
run: echo "${{ secrets.gerritCookie }}" > ~/.gitcookies
- name: Install Go
uses: actions/setup-go@v2
with:
Expand All @@ -38,7 +40,7 @@ jobs:
key: ${{ runner.os }}-${{ matrix.go-version }}-go-${{ hashFiles('**/go.sum')
}}
restore-keys: ${{ runner.os }}-${{ matrix.go-version }}-go-
- if: matrix.go-version == '1.14.9' && matrix.os != 'windows-latest'
- if: matrix.go-version == '1.14.9' && matrix.os != 'windows-2019'
name: Generate
run: go generate ./...
- name: Test
Expand All @@ -49,10 +51,41 @@ jobs:
run: go run golang.org/x/exp/cmd/gorelease
- name: Check that git is clean post generate and tests
run: test -z "$(git status --porcelain)" || (git status; git diff; false)
- if: github.ref == 'refs/heads/master'
- if: ${{ github.ref == 'refs/heads/master' }}
name: Pull this commit through the proxy on master
run: |-
v=$(git rev-parse HEAD)
cd $(mktemp -d)
go mod init mod.com
GOPROXY=https://proxy.golang.org go get -d cuelang.org/go@$v
- if: ${{ startsWith(github.ref, 'refs/heads/ci/') && failure() }}
name: Post any failures for this matrix entry
run: 'curl -f -s -H "Content-Type: application/json" --request POST --data ''{"message":"Build
failed for ${{ runner.os }}-${{ matrix.go-version }}; see ${{ github.event.repository.html_url
}}/actions/runs/${{ github.run_id }} for more details","labels":{"Code-Review":-1}}''
-b ~/.gitcookies https://cue-review.googlesource.com/a/changes/$(basename
$(dirname $GITHUB_REF))/revisions/$(basename $GITHUB_REF)/review'
mark_ci_success:
runs-on: ubuntu-18.04
if: ${{ startsWith(github.ref, 'refs/heads/ci/') }}
needs: test
defaults:
run:
shell: bash
steps:
- name: Write the gitcookies file
run: echo "${{ secrets.gerritCookie }}" > ~/.gitcookies
- name: Update Gerrit CL message with success message
run: 'curl -f -s -H "Content-Type: application/json" --request POST --data ''{"message":"Build
succeeded for ${{ github.event.repository.html_url }}/actions/runs/${{ github.run_id
}}","labels":{"Code-Review":1}}'' -b ~/.gitcookies https://cue-review.googlesource.com/a/changes/$(basename
$(dirname $GITHUB_REF))/revisions/$(basename $GITHUB_REF)/review'
delete_build_branch:
runs-on: ubuntu-18.04
if: ${{ startsWith(github.ref, 'refs/heads/ci/') && always() }}
needs: test
defaults:
run:
shell: bash
steps:
- run: echo git push origin :${GITHUB_REF#refs/heads/}
79 changes: 11 additions & 68 deletions .github/workflows/test_dispatch.yml
Original file line number Diff line number Diff line change
@@ -1,86 +1,29 @@
# Generated by internal/ci/ci_tool.cue; do not edit

name: Test Dispatch
env:
GERRIT_COOKIE: ${{ secrets.gerritCookie }}
on:
- repository_dispatch
defaults:
run:
shell: bash
jobs:
start:
runs-on: ubuntu-latest
if: ${{ startsWith(github.event.action, 'Build for refs/changes/') }}
runs-on: ubuntu-18.04
defaults:
run:
shell: bash
steps:
- name: Write the gitcookies file
run: echo "$GERRIT_COOKIE" > ~/.gitcookies
run: echo "${{ secrets.gerritCookie }}" > ~/.gitcookies
- name: Update Gerrit CL message with starting message
run: 'curl -f -s -H "Content-Type: application/json" --request POST --data ''{"message":"Started
the build... see progress at ${{ github.event.repository.html_url }}/actions/runs/${{
github.run_id }}"}'' -b ~/.gitcookies https://cue-review.googlesource.com/a/changes/${{
github.event.client_payload.changeID }}/revisions/${{ github.event.client_payload.commit
}}/review'
test:
runs-on: ${{ matrix.os }}
steps:
- name: Write the gitcookies file
run: echo "$GERRIT_COOKIE" > ~/.gitcookies
- name: Install Go
uses: actions/setup-go@v2
with:
go-version: ${{ matrix.go-version }}
- name: Checkout code
uses: actions/checkout@v2
- name: Checkout ref
run: |-
git config --global user.name cueckoo
git config --global user.email cueckoo@gmail.com
git config http.https://github.com/.extraheader "AUTHORIZATION: basic $(echo -n cueckoo:${{ secrets.CUECKOO_GITHUB_PAT }} | base64)"
git fetch https://cue-review.googlesource.com/cue ${{ github.event.client_payload.ref }}
git checkout FETCH_HEAD
- name: Cache Go modules
uses: actions/cache@v1
with:
path: ~/go/pkg/mod
key: ${{ runner.os }}-${{ matrix.go-version }}-go-${{ hashFiles('**/go.sum')
}}
restore-keys: ${{ runner.os }}-${{ matrix.go-version }}-go-
- if: matrix.go-version == '1.14.9' && matrix.os != 'windows-latest'
name: Generate
run: go generate ./...
- name: Test
run: go test ./...
- name: Test with -race
run: go test -race ./...
- name: gorelease check
run: go run golang.org/x/exp/cmd/gorelease
- name: Check that git is clean post generate and tests
run: test -z "$(git status --porcelain)" || (git status; git diff; false)
- if: ${{ failure() }}
name: Post any failures for this matrix entry
run: 'curl -f -s -H "Content-Type: application/json" --request POST --data ''{"message":"Build
failed for ${{ runner.os }}-${{ matrix.go-version }}; see ${{ github.event.repository.html_url
}}/actions/runs/${{ github.run_id }} for more details","labels":{"Code-Review":-1}}''
-b ~/.gitcookies https://cue-review.googlesource.com/a/changes/${{ github.event.client_payload.changeID
}}/revisions/${{ github.event.client_payload.commit }}/review'
needs: start
strategy:
fail-fast: false
matrix:
go-version:
- 1.13.x
- 1.14.9
- 1.15.x
os:
- ubuntu-latest
- macos-latest
- windows-latest
end:
runs-on: ubuntu-latest
steps:
- name: Write the gitcookies file
run: echo "$GERRIT_COOKIE" > ~/.gitcookies
- name: Update Gerrit CL message with success message
run: 'curl -f -s -H "Content-Type: application/json" --request POST --data ''{"message":"Build
succeeded for ${{ github.event.repository.html_url }}/actions/runs/${{ github.run_id
}}","labels":{"Code-Review":1}}'' -b ~/.gitcookies https://cue-review.googlesource.com/a/changes/${{
github.event.client_payload.changeID }}/revisions/${{ github.event.client_payload.commit
}}/review'
needs: test
git checkout -b ci/${{ github.event.client_payload.changeID }}/${{ github.event.client_payload.commit }} FETCH_HEAD
git push origin ci/${{ github.event.client_payload.changeID }}/${{ github.event.client_payload.commit }}
Loading

0 comments on commit ccbfbd2

Please sign in to comment.