Skip to content

Commit

Permalink
Reenable JMH benchmarking in a safer manner (#975)
Browse files Browse the repository at this point in the history
Fixes #968 

The key difference is now the benchmarking job only starts when the
label with the name `run-benchmarks` is added to a PR. According to [the
docs](https://docs.github.com/en/issues/using-labels-and-milestones-to-track-work/managing-labels#applying-a-label)
only those with triage access to the repository can add or remove a
label. In contrast, anyone can comment on an issue, which made our
previous technique for kick-starting the benchmarks unsafe. Before
adding the `run-benchmarks` label, a PR should be reviewed to check for
malicious code.

It is impossible to test this workflow without first merging it to the
main branch. However, I did test it using a PR on my fork, and confirmed
it could comment back the benchmark results like before:

msridhar#12

After merging this PR we'll also have to add some credentials to the
main NullAway repo to make this work. But first we should review, land,
and then see that it fails as expected without the credentials.

Compared to [the earlier workflow
file](https://github.com/uber/NullAway/blob/87ec10d4f26630d4bb91aefe5ff7c0fc181f030a/.github/workflows/jmh-benchmark.yml),
beyond changing to use labeling, I updated the versions of some external
GitHub actions we are using.
  • Loading branch information
msridhar authored Jun 12, 2024
1 parent 87f16cd commit ab5972a
Showing 1 changed file with 68 additions and 0 deletions.
68 changes: 68 additions & 0 deletions .github/workflows/jmh-benchmark.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# This GitHub Actions workflow runs JMH benchmarks when requested
name: Run JMH Benchmarks for Pull Request

on:
pull_request: # Trigger when a label is added. Only contributors have permissions to add labels
types: [labeled]

# Only allow one instance of JMH benchmarking to be running at any given time
concurrency: all

jobs:
benchmarking:
# Only run this job if the label is 'run-benchmarks' and the PR in on the uber/NullAway repository
if: github.event.label.name == 'run-benchmarks' && github.repository == 'uber/NullAway'
runs-on: ubuntu-latest
permissions: write-all

steps:

- name: Checkout repository
uses: actions/checkout@v4

- name: Set branch name
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
chmod +x ./.github/workflows/get_repo_details.sh
./.github/workflows/get_repo_details.sh "${{ secrets.GITHUB_TOKEN }}" "${{ github.event.number }}" "${{ github.repository }}"
- id: 'auth'
name: Authenticating
uses: google-github-actions/auth@v2
with:
credentials_json: '${{ secrets.GCP_SA_KEY_1 }}'

- name: Set up Google Cloud SDK
uses: google-github-actions/setup-gcloud@v2

- name: Start VM
run: gcloud compute instances start nullway-jmh --zone=us-central1-a

- name: Run benchmarks
run: |
chmod +x ./.github/workflows/run_gcp_benchmarks.sh
./.github/workflows/run_gcp_benchmarks.sh
- name: Cleanup
# Delete the branch directory on the Google Cloud instance
if: always()
run: |
./.github/workflows/gcloud_ssh.sh " export BRANCH_NAME=${BRANCH_NAME} && rm -r -f $BRANCH_NAME"
- name: Formatting Benchmark # Create a text file containing the benchmark results
run: |
(echo 'Main Branch:'; echo '```' ; cat main_text.txt; echo '```'; echo 'With This PR:'; echo '```' ; cat pr_text.txt; echo '```') > benchmark.txt
- name: Comment Benchmark
uses: mshick/add-pr-comment@v2
if: always() # This step is for adding the comment
with:
message-path: benchmark.txt # The path to the message file to leave as a comment
message-id: benchmark
- name: Stop VM
if: always()
run: gcloud compute instances stop nullway-jmh --zone=us-central1-a



0 comments on commit ab5972a

Please sign in to comment.