Skip to content

Commit

Permalink
Merge pull request #5245 from Sage-Bionetworks/SWC-6624
Browse files Browse the repository at this point in the history
SWC-6624: store Playwright reports with traces in S3 when e2e workflow runs in Sage repo
  • Loading branch information
hallieswan authored Dec 13, 2023
2 parents 629b3b7 + 18288b0 commit 5b74853
Show file tree
Hide file tree
Showing 5 changed files with 168 additions and 7 deletions.
39 changes: 34 additions & 5 deletions .github/workflows/build-test-e2e.yml
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@ env:
PW_ALL_BLOBS_DIR: all-blob-reports
jobs:
build:
# Run in Sage repo on release- branches
# Run in Sage repo on develop or release- branches
# and on all branches in user-owned forks
if: ${{ startsWith(github.ref_name, 'release-') || github.actor == github.repository_owner }}
if: ${{ github.ref_name == 'develop' || startsWith(github.ref_name, 'release-') || github.actor == github.repository_owner }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
Expand All @@ -27,6 +27,16 @@ jobs:
needs: [build]
runs-on: 'macos-latest'
timeout-minutes: 60
# Ensure that at most one playwright-tests job will run at a time in Sage-Bionetworks repo,
# but allow multiple playwright-test jobs to run concurrently in forked repos
# Per https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#concurrency
# "When a concurrent job or workflow is queued, if another job or workflow
# using the same concurrency group in the repository is in progress,
# the queued job or workflow will be pending. Any previously pending job or
# workflow in the concurrency group will be cancelled."
# Related discussion here: https://github.com/orgs/community/discussions/5435
concurrency:
group: ${{ github.repository_owner == 'Sage-Bionetworks' && format('${0}-${1}', github.workflow, '-playwright-tests') || format('${0}-${1}', github.run_id, matrix.shard ) }}
strategy:
max-parallel: ${{ github.repository_owner == 'Sage-Bionetworks' && 1 || 3 }}
fail-fast: false
Expand All @@ -35,6 +45,12 @@ jobs:
env:
CONTAINER_NAME: 'swc-tomcat'
CI: true
DESTINATION_BUCKET: e2e-reports-bucket-bucket-1p1qz6p48t4uy
REPORT_ID: ${{ github.event.repository.name }}-${{ github.run_id }}-${{ github.run_attempt }}
# Allow OIDC Integration (so we can assume the AWS role to deploy)
permissions:
id-token: write
contents: read
steps:
- uses: actions/checkout@v3
- name: Setup docker (missing on MacOS)
Expand Down Expand Up @@ -66,10 +82,23 @@ jobs:
- name: Run Playwright tests
env:
ADMIN_PAT: ${{ secrets.ADMIN_PAT }}
run: yarn playwright test --shard ${{ matrix.shard }}
TRACE_TOGGLE: ${{ github.repository_owner == 'Sage-Bionetworks' && 'on' || 'off'}}
run: yarn playwright test --shard ${{ matrix.shard }} --trace=${{ env.TRACE_TOGGLE }}
- name: Assume AWS Role
if: github.repository_owner == 'Sage-Bionetworks'
uses: aws-actions/configure-aws-credentials@v2
with:
aws-region: us-east-1
role-to-assume: arn:aws:iam::449435941126:role/sagebase-github-oidc-synapsedev-e2e-infra
role-session-name: GitHubActions-${{ github.repository_owner }}-${{ github.event.repository.name }}-${{ github.run_id }}
role-duration-seconds: 1200
- name: Upload blob report to S3
if: github.repository_owner == 'Sage-Bionetworks'
run: |
aws s3 sync ./blob-report --region us-east-1 s3://${{ env.DESTINATION_BUCKET }}/${{ env.REPORT_ID }}/
- name: Upload blob report to GitHub Actions Artifacts
uses: actions/upload-artifact@v3
if: always()
if: ${{ github.repository_owner != 'Sage-Bionetworks' && always() }}
with:
name: ${{ env.PW_ALL_BLOBS_DIR }}
path: blob-report
Expand All @@ -83,7 +112,7 @@ jobs:
merge-reports:
# Merge reports after playwright-tests, even if some shards have failed
# But skip this job if the previous job was cancelled or skipped
if: ${{ !cancelled() && needs.playwright-tests.result != 'skipped' }}
if: ${{ github.repository_owner != 'Sage-Bionetworks' && !cancelled() && needs.playwright-tests.result != 'skipped' }}
needs: [playwright-tests]
runs-on: ubuntu-latest
steps:
Expand Down
3 changes: 2 additions & 1 deletion .prettierignore
Original file line number Diff line number Diff line change
Expand Up @@ -49,5 +49,6 @@ src/main/webapp/fonts/*
# If using jenv for JDK management
/.java-version

# Ignore e2e mock data files
# Ignore e2e mock data files and scripts
/e2e/data/*.csv
/e2e_workflow/*.sh
36 changes: 35 additions & 1 deletion e2e_workflow/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,41 @@ Notes:

### CI

In your forked repository, ensure that [Actions are enabled](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository), then create [an Actions secret](https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository) for `ADMIN_PAT`. The tests will run on each push to your forked repository.
Playwright reports are configured and saved differently based on where the e2e workflow runs. In forked repos, reports are saved as GitHub artifacts, which are publically accessible in public repositories. Since traces can include credentials for the backend dev stack, these reports do not include traces. In the `Sage-Bionetworks`-owned repo, Playwright reports are saved to a private S3 bucket so the reports can include traces.

#### Forked Repositories

In your forked repository, ensure that [Actions are enabled](https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository), then create [an Actions secret](https://docs.github.com/en/actions/security-guides/encrypted-secrets#creating-encrypted-secrets-for-a-repository) for `ADMIN_PAT`. The tests will run on each push to your forked repository. The Playwright report will be saved as a GitHub Artifact.

The GitHub UI or CLI can be used to view the reports:

- GitHub UI
- Navigate to the Action run summary page.
- Download the report named "html-report--attempt-{number}". _Note:_ only the report from the latest attempt will be available.
- Unzip the file and move the "index.html" file into the `playwright-report` directory in SWC.
- Run `yarn e2e:report` to view the HTML report in the browser.
- GitHub CLI
- Install [GitHub CLI](https://cli.github.com/), if necessary.
- Install [jq](https://jqlang.github.io/jq/download/), if necessary.
- [Authenticate](https://cli.github.com/manual/gh_auth_login) with a GitHub host, if necessary.
- Run `e2e_workflow/view_github_report.sh` with the following arguments: GitHub repo owner name and the GitHub run ID of the report to view. The HTML report will open in the browser.

#### Sage Repository

In the `Sage-Bionetworks`-owned repository, the tests will run on push to `develop` and `release-**` branches. The Playwright blob report from each shard will be saved in the `s3://e2e-reports-bucket-bucket-1p1qz6p48t4uy` S3 bucket. Each report will be named as `report-{ shard-number }.zip` and be located within a subdirectory named `SynapseWebClient-${ github-run-id }-${ run-attempt-number }`.

The AWS console or CLI can be used to view the reports:

- AWS console
- Authenticate with AWS console SSO with the `org-sagebase-synapsedev` account.
- Navigate to the S3 bucket: `s3://e2e-reports-bucket-bucket-1p1qz6p48t4uy`
- Download the shard reports locally.
- Move the files into the `blob-report` directory in SWC.
- Run `yarn e2e:report:blob` to merge the shard reports and open the resulting HTML report in the browser.
- AWS CLI
- Install [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html), if necessary.
- Configure access credentials for AWS CLI SSO with the `org-sagebase-synapsedev` account.
- Run `e2e_workflow/view_s3_report.sh` with the following arguments: AWS SSO profile name, the GitHub run ID of the report to view, and optionally, the run attempt number. The HTML report will open in the browser.

## Writing Tests

Expand Down
57 changes: 57 additions & 0 deletions e2e_workflow/view_github_report.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
#!/bin/bash

# View Playwright report stored in GitHub
# ./view_github_report.sh [github-repo-owner] [github-run-id]
#

# Abort script on errors and unbound variables
# https://bertvv.github.io/cheat-sheets/Bash.html
set -o errexit # abort on nonzero exitstatus
set -o nounset # abort on unbound variable
set -o pipefail # don't hide errors within pipes

# Set variables
SCRIPT_PATH="$(cd "$(dirname "${0}")"; echo $(pwd))"
REPO_PATH="$(dirname ${SCRIPT_PATH})"
REPO_NAME="$(basename ${REPO_PATH})"
BLOB_DIR="${REPO_PATH}/blob-report/"
REPORT_DIR="${REPO_PATH}/playwright-report/"

# Get parameters from user
if [ "$#" -lt 2 ]; then
echo "Usage: $0 [github-repo-owner] [github-run-id]"
echo "* github-repo-owner: GitHub repo owner"
echo "* github-run-id: GitHub Run ID of report to view"
exit
fi
REPO_OWNER="${1}"
RUN_ID="${2}"

# Delete previous report files
for file in $(find "${BLOB_DIR}" -type f) $(find "${REPORT_DIR}" -type f); do
[ -f "${file}" ] && echo "delete: ${file}" && rm "${file}"
done

# Get workflow artifact id and name
ARTIFACT=$(gh api \
-H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
"/repos/${REPO_OWNER}/${REPO_NAME}/actions/runs/${RUN_ID}/artifacts" \
| jq '.artifacts[] | select(.name | contains("html-report")) | {id, name}')
ARTIFACT_ID=$(echo "${ARTIFACT}" | jq '.id')
ARTIFACT_NAME=$(echo "${ARTIFACT}" | jq -r '.name')

# Download workflow artifact
ARTIFACT_ENDPOINT="/repos/${REPO_OWNER}/${REPO_NAME}/actions/artifacts/${ARTIFACT_ID}/zip"
ARTIFACT_ZIP="${BLOB_DIR}/${ARTIFACT_NAME}.zip"
echo "download: ${ARTIFACT_ENDPOINT} to ${ARTIFACT_ZIP}"
gh api \
-H "Accept: application/vnd.github+json" \
-H "X-GitHub-Api-Version: 2022-11-28" \
"${ARTIFACT_ENDPOINT}" > "${ARTIFACT_ZIP}"

# Unzip file
unzip "${ARTIFACT_ZIP}" -d "${REPORT_DIR}"

# Show report
yarn e2e:report
40 changes: 40 additions & 0 deletions e2e_workflow/view_s3_report.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
#!/bin/bash

# View Playwright report stored in S3
# ./view_s3_report.sh [aws-sso-profile-name] [github-run-id] [optional-run-attempt-number]
#

# Abort script on errors and unbound variables
# https://bertvv.github.io/cheat-sheets/Bash.html
set -o errexit # abort on nonzero exitstatus
set -o nounset # abort on unbound variable
set -o pipefail # don't hide errors within pipes

# Set variables
BUCKET_NAME="e2e-reports-bucket-bucket-1p1qz6p48t4uy"
SCRIPT_PATH="$(cd "$(dirname "${0}")"; echo $(pwd))"
REPO_PATH="$(dirname ${SCRIPT_PATH})"
REPO_NAME="$(basename ${REPO_PATH})"
BLOB_DIR="${REPO_PATH}/blob-report/"

# Get parameters from user
if [ "$#" -lt 2 ]; then
echo "Usage: $0 [aws-sso-profile-name] [github-run-id] [optional-run-attempt-number]"
echo "* aws-sso-profile-name: AWS SSO profile name, e.g. Developer-{number}"
echo "* github-run-id: GitHub Run ID of report to view"
echo "* optional-run-attempt-number: (optional) Run attempt number. Defaults to 1."
exit
fi
PROFILE="${1}"
RUN_ID="${2}"
RUN_ATTEMPT="${3:-1}"

# Sync blob reports
aws s3 sync \
s3://"${BUCKET_NAME}"/"${REPO_NAME}-${RUN_ID}-${RUN_ATTEMPT}"/ \
"${BLOB_DIR}" \
--profile "${PROFILE}" \
--delete

# Merge blob reports, then serve as HTML report
yarn e2e:report:blob

0 comments on commit 5b74853

Please sign in to comment.