Skip to content

Commit

Permalink
feat(ci): analyses snapshot skip pr on fail label (#15792)
Browse files Browse the repository at this point in the history
## Overview

- ~~Keep making the analyses snapshot test workflow more flexible
allowing a label on the PR to prevent failure PRs from opening. Adding
the label `no-analyses-snapshot-pr` on your PR will cause the workflow
to skip opening a PR for snapshot updates if the snapshot tests fail.~~
- Added a step to diff the snapshots in the PR target branch against the
PR branch. No action is taken based on the outcome. Purely
informational. Will this information be noticed and used as a reminder
to synchronize with the target branch?

### Updates from feedback

- default `OPEN_PR_ON_FAILURE` to false 
- as is the case currently if the analyses snapshot test fails you will
see a red X on that test and you can read the logs or download and view
the HTML test report.
- if you want to open a PR into your branch that would heal the
snapshots, add `gen-analyses-snapshot-pr` label and a PR will get opened
- create a specific step to handle PRs for the overnight scheduled test
of edge.

### Risk

No risk or change to application code, CI only
  • Loading branch information
y3rsh authored Jul 25, 2024
1 parent d8f8023 commit eaf9657
Showing 1 changed file with 40 additions and 9 deletions.
49 changes: 40 additions & 9 deletions .github/workflows/analyses-snapshot-test.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -23,8 +23,16 @@ on:
- 'api/**'
- '!api/tests/**'
- '!api/docs/**'
- '!api/release-notes-internal.md'
- '!api/release-notes.md'
- 'shared-data/**/*'
- '!shared-data/js/**'
- '.github/workflows/analyses-snapshot-test.yaml'
types:
- opened #default
- synchronize #default
- reopened #default
- labeled

concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
Expand All @@ -40,14 +48,27 @@ jobs:
# If we're running because of workflow_dispatch, use the user input to decide
# whether to open a PR on failure. Otherwise, there is no user input, so always
# open a PR on failure.
OPEN_PR_ON_FAILURE: ${{ (github.event_name == 'workflow_dispatch' && github.events.inputs.OPEN_PR_ON_FAILURE) || (github.event_name != 'workflow_dispatch') }}

OPEN_PR_ON_FAILURE: ${{ (github.event_name == 'workflow_dispatch' && github.events.inputs.OPEN_PR_ON_FAILURE) || ((github.event_name != 'workflow_dispatch') && (contains(github.event.pull_request.labels.*.name, 'gen-analyses-snapshot-pr'))) }}
PR_TARGET_BRANCH: ${{ github.event.pull_request.base.ref || 'not a pr'}}
steps:
- name: Checkout Repository
uses: actions/checkout@v4
with:
ref: ${{ env.SNAPSHOT_REF }}

- name: Are the analyses snapshots in my PR branch in sync with the target branch?
if: github.event_name == 'pull_request'
run: |
git fetch origin ${{ env.PR_TARGET_BRANCH }}
DIFF_OUTPUT=$(git diff HEAD origin/${{ env.PR_TARGET_BRANCH }} -- analyses-snapshot-testing/tests/__snapshots__/analyses_snapshot_test)
if [ -n "$DIFF_OUTPUT" ]; then
echo "Analyses snapshots do NOT match ${{ env.PR_TARGET_BRANCH }} snapshots."
echo "Is this becasue you have not pulled and merged ${{ env.PR_TARGET_BRANCH }}?"
echo "Or is this because you have already updated your snapshots and are all good 😊?"
else
echo "Analyses snapshots match ${{ env.PR_TARGET_BRANCH }} snapshots."
fi
- name: Docker Build
working-directory: analyses-snapshot-testing
run: make build-opentrons-analysis
Expand Down Expand Up @@ -77,30 +98,40 @@ jobs:

- name: Handle Test Failure
id: handle_failure
if: always() && steps.run_test.outcome == 'failure'
if: always() && steps.run_test.outcome == 'failure' && (env.OPEN_PR_ON_FAILURE == 'true' || github.event_name == 'schedule')
working-directory: analyses-snapshot-testing
run: make snapshot-test-update

- name: Create Snapshot update Request
id: create_pull_request
if: always() && steps.handle_failure.outcome == 'success' && env.OPEN_PR_ON_FAILURE
if: always() && steps.handle_failure.outcome == 'success' && env.OPEN_PR_ON_FAILURE == 'true' && github.event_name == 'pull_request'
uses: peter-evans/create-pull-request@v6
with:
commit-message: 'fix(analyses-snapshot-testing): snapshot failure capture'
title: 'fix(analyses-snapshot-testing): ${{ env.ANALYSIS_REF }} snapshot failure capture'
body: 'This PR is an automated snapshot update request. Please review the changes and merge if they are acceptable or find your bug and fix it.'
commit-message: 'fix(analyses-snapshot-testing): heal analyses snapshots'
title: 'fix(analyses-snapshot-testing): heal ${{ env.ANALYSIS_REF }} snapshots'
body: 'This PR was requested on the PR https://github.com/${{ github.repository }}/pull/${{ github.event.pull_request.number }}'
branch: 'analyses-snapshot-testing/${{ env.ANALYSIS_REF }}-from-${{ env.SNAPSHOT_REF}}'
base: ${{ env.SNAPSHOT_REF}}

- name: Comment on PR
- name: Comment on feature PR
if: always() && steps.create_pull_request.outcome == 'success' && github.event_name == 'pull_request'
uses: actions/github-script@v7
with:
script: |
const message = 'A PR has been opened to address analyses snapshot changes. Please review the changes here: https://github.com/${{ github.repository }}/pull/${{ steps.create-pull-request.outputs.pull-request-number }}';
const message = 'A PR has been opened to address analyses snapshot changes. Please review the changes here: https://github.com/${{ github.repository }}/pull/${{ steps.create_pull_request.outputs.pull-request-number }}';
github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
body: message
});
- name: Create Snapshot update Request on edge overnight failure
if: always() && steps.handle_failure.outcome == 'success' && github.event_name == 'schedule'
uses: peter-evans/create-pull-request@v6
with: # scheduled run uses the default values for ANALYSIS_REF and SNAPSHOT_REF which are edge
commit-message: 'fix(analyses-snapshot-testing): heal ${{ env.ANALYSIS_REF }} snapshots'
title: 'fix(analyses-snapshot-testing): heal ${{ env.ANALYSIS_REF }} snapshots'
body: 'The ${{ env.ANALYSIS_REF }} overnight analyses snapshot test is failing. This PR was opened to alert us to the failure.'
branch: 'analyses-snapshot-testing/${{ env.ANALYSIS_REF }}-from-${{ env.SNAPSHOT_REF}}'
base: ${{ env.SNAPSHOT_REF}}

0 comments on commit eaf9657

Please sign in to comment.