Skip to content

Commit

Permalink
Create reusable workflow for E2E tests (#13883)
Browse files Browse the repository at this point in the history 
* wip nightly pipeline

add nightly pipeline
add check tool that will be executed in PRs if a test from integration-tests is added or modified. It will report a comment or block pipeline requiring test to be added to the nightly workflow

* add reusable workflow

* on push

* echo tests

* fix matrix

* fix

* add job name

* run docker tests

* add reusable workflow

* fix

* fail if no test found

* fix run selected workflow

* Rename reusable workflow

* Add nighly workflow

* try running k8s tests

* cleanup

* add permissions

* fix

* test

* Fix

* fix

* test remote runner

* Add test suite param

* Run all_tests_in_ocr_tests_go in parallel

* Update e2e-tests.yml

* Add check-tests cmd

* Unify tools into single cmd

* Add check tests step to workflow

* show more info in annotation

* fix

* Do not require check-tests

* fix go mod

* temp: Disable checking test configuration

* Add enable-check-test-configurations to reusable workflow. False by default

* Add automation upgrade test to e2e list

* Fix enable-check-test-configurations

* Fix test cmd

* Make enable-check-test-configurations false by default

* support test config override

* add more automation tests

* Run automation upgrade tests on latest develop

* Migrate automation-nightly-tests.yml to reusable workflow

* Update automation tests

* Fix automation workflow

* Fix

* Fix

* Fx

* Fix workflow_call inputs

* Select tests by github workflows

* Allow either test-ids or test-workflow in reusable workflow

* Fix

* Update workflow

* Add comment

* Fix

* Fix

* Update e2e tests

* Add remote-runner-memory option

* Disable nightly run for now

* Fix output in k8s tests

* Add show test configuration step

* Uncomment building test runner image

* Do not use json output for k8s tests

* test

* Support custom runner version

* Fix test type env in remote runner

* Update names and docs

* Add script to prepare test config override

* Add CI test config override tools

* Add mask-secrets cmd and stop using reusable workflow

* Use reusable workflow in run-selected-e2e-tests.yml

* Remove unnecessary permissions

* Add a sample test to quickly check CI workflows

* Do not use mask cmd in reusable workflow

* Clean up override test config cmd

* Fix masking of full chainlink image

* Finish "Set default test config override" step in reusable workflow

* Remove env.CHAINLINK_IMAGE mask as it is not a scret

* Fix reusable workflow

* Create create-default-e2e-config-override action

* Fix

* Fix log targets in GHA and add test

* Add create-default-e2e-config step to k8s tests

* Fix ocr tests

* Use sanitized id for artifact name

* Update nightly workflow to test it

* Fail CI if test_config_base64_override is used

* Use underscore for github inputs

* Fix

* Fix inputs.chainlink_version

* Remove test_config_base64_override from inputs until we are sure it does not have secrets

This will be uncommented when https://smartcontract-it.atlassian.net/browse/TT-1283 is done

* Fix

* Remove unused action

* Fix

* Update create config override cmd and action

* Refactor client-compatibility-tests.yml to use actions/create-default-e2e-config-override

* Refactor on-demand-keeper-smoke-tests.yml

* Replace setup-create-base64-config with the new action in integration-tests.yml workflow

* Remove unused action

* Fix lint

* Remove unused action

* Fix handling selected_networks input

* Fix cache

* Do not run soak/ocr_test.go:^TestOCRSoak$ nightly

* Enable nightly workflow to run some tests

* Remove config override input

* Remove temp test

* Fix setup go

* go mod tidy in core/scripts

* Fix

* Remove unready workflows

* Revert updates to core/scripts

* Revert updates to go.mod

* Remove unused script

* Add setup-gap id

* Remove unused steps

* test nightly workflow

* Add missing env vars in remote runner and fix lack of logs

* bump setup gap

* Fix set-output

* update comment

* update test name

* update cache

* Fix empty output for k8s tests

* Add a test workflow with overrides

* Fix cache

* Resolve env vars in filter cmd

* Fix

* Rename test overrides to test inputs

* Update input descriptions

* Fix

* Check chainlink image exist in automation ondemand workflow

* Add require-chainlink-image-versions-in-qa-ecr job to reusable workflow

* Set no_cache for run-tests action

* Fix

* Add test_secrets_override_key to run-selected-tests workflow

* Remove chainlink image and version from create test config cmd

* Update log

* Remove chainlik image from create config tool

* Remove chainlik image from create config tool

* Set default upgrade version for config validation to pass when upgrade image set in secrets

* bump gha

* Fix

* for test

* bump gha

* Move e2e ci tool to CTF

* bump run-tests gha

* bump ctf

* Allow to pass custom base64 test config override to reusable workflow

* Update e2e-tests.yml

* Use DEFAULT_CHAINLINK_UPGRADE_IMAGE in workflow

* Add run-name

* fix custom base64

* Fix workflow name

* Fix test inputs

* bump

* Remove default chainlink_version

* Bump

* Use CTF with no secret keys in toml

* add soak/ocr_test.go:^TestOCRv2Soak$

* update e2e tests

* Fix setting pyroscope defaults

* Allow custom test secrets in on-demand-ocr-soak-test.yml

* Bump CTF to mount test secrets in remote runner

* bump to use configmap

* bump ctf

* bump

* Use ctf latest release

* Add test_secrets_override_base64 to other workflows that have testConfigBase64 input

* Do not fail workflow if no chainlink image found in setup-parse-base64-config

* Bump gha

* Test test-config-override-required

* Bump CTF

* Bump ctf

* Add integration-tests/load/vrfv2plus/vrfv2plus_test.go:^TestVRFV2PlusPerformance$

* Bump ctf

* Fix

* Update test suite

* Add vrf tests to e2e-tests.yml

* Cleanup

* Add integration-tests/load/automationv2_1/automationv2_1_test.go to list of tests

* Add rest of ocr soak tests

* Add integration-tests/migration/upgrade_version_test.go:*

* Use snake case

* Do not set default chainlink_upgrade_version to pass config validation

* Use develop-plugins as default version for plugins tests

* Fix

* Add automation and keeper smoke tests

* Add log poller test

* Add more log poller tests

* Clean up e2e tests

* Fix

* Fix list of tests

* Prioritise chainlink_version in test_inputs before the workflow input

* Fix

* Move citool from CTF to Core repo

* Fix import

* Fix

* Fix check-tests cmd

* bump run-tests

* Add slack notification after tests

* Add collect metrics step to reusable workflow

* Fix tag

* Fix job name in collect metrics steps

* bump ctf

* fix collect metrics

* Disallow passing test_config_override_base64 until TT-1392 is done

* Bump run-tests GHA

* Update channel for slack notification for nightly tests

* Fix run-automation-ondemand-e2e-tests.yml

* Do not show base64 config override

* Fix lint

* Fix config flag

* Remove unused cmd

* Fix go mod

* Fix concurrent map writes

https://github.com/smartcontractkit/chainlink/actions/runs/9988450051/job/27605332768

* Flip cache

* Update automation workflow

* revert MOD_CACHE_VERSION

* Remove image keys from toml in automation-ondemand-tests.yml

* Bump run-test GHA

* Fix reorg workflow according to Anirudh suggestion

* Fix

* Fix automation-ondemand-tests.yml

* Bump run-tests GHA in automation-ondemand-tests.yml
  • Loading branch information
@lukaszcl
lukaszcl authored Jul 18, 2024
1 parent 3782a9c commit 901609e
Show file tree
Hide file tree
Showing 38 changed files with 2,920 additions and 277 deletions.
106 changes: 0 additions & 106 deletions .github/actions/setup-create-base64-config/action.yml
View file

This file was deleted.

10 changes: 4 additions & 6 deletions .github/actions/setup-merge-base64-config/action.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -24,22 +24,20 @@ runs:
echo "BASE64_CONFIG_OVERRIDE=$BASE64_CONFIG_OVERRIDE" >> $GITHUB_ENV
decoded_toml=$(echo $BASE64_CONFIG_OVERRIDE | base64 -d)
CHAINLINK_IMAGE=$(echo "$decoded_toml" | dasel -r toml 'ChainlinkImage.image')
CHAINLINK_IMAGE=$(echo "$decoded_toml" | { dasel -r toml 'ChainlinkImage.image' 2>/dev/null || echo ''; })
echo ::add-mask::$CHAINLINK_IMAGE
CHAINLINK_VERSION=$(echo "$decoded_toml" | dasel -r toml 'ChainlinkImage.version')
CHAINLINK_VERSION=$(echo "$decoded_toml" | { dasel -r toml 'ChainlinkImage.version' 2>/dev/null || echo ''; })
NETWORKS=$(echo "$decoded_toml" | awk -F'=' '/^[[:space:]]*selected_networks[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); print $2}' 2>/dev/null)
if [ -n "$CHAINLINK_IMAGE" ]; then
echo "CHAINLINK_IMAGE=$CHAINLINK_IMAGE" >> $GITHUB_ENV
else
echo "No Chainlink Image found in base64-ed config. Exiting"
exit 1
echo "No Chainlink Image found in base64-ed config"
fi
if [ -n "$CHAINLINK_VERSION" ]; then
echo "CHAINLINK_VERSION=$CHAINLINK_VERSION" >> $GITHUB_ENV
else
echo "No Chainlink Version found in base64-ed config. Exiting"
exit 1
echo "No Chainlink Version found in base64-ed config"
fi
if [ -n "$NETWORKS" ]; then
echo "NETWORKS=$NETWORKS" >> $GITHUB_ENV
Expand Down
8 changes: 3 additions & 5 deletions .github/actions/setup-parse-base64-config/action.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -20,23 +20,21 @@ runs:
shell: bash
run: |
decoded_toml=$(echo $BASE64_CONFIG_OVERRIDE | base64 -d)
CHAINLINK_IMAGE=$(echo "$decoded_toml" | dasel -r toml 'ChainlinkImage.image')
CHAINLINK_IMAGE=$(echo "$decoded_toml" | { dasel -r toml 'ChainlinkImage.image' 2>/dev/null || echo ''; })
echo ::add-mask::$CHAINLINK_IMAGE
CHAINLINK_VERSION=$(echo "$decoded_toml" | dasel -r toml 'ChainlinkImage.version')
CHAINLINK_VERSION=$(echo "$decoded_toml" | { dasel -r toml 'ChainlinkImage.version' 2>/dev/null || echo ''; })
NETWORKS=$(echo "$decoded_toml" | awk -F'=' '/^[[:space:]]*selected_networks[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); print $2}' 2>/dev/null)
ETH2_EL_CLIENT=$(echo "$decoded_toml" | awk -F'=' '/^[[:space:]]*execution_layer[[:space:]]*=/ {gsub(/^[[:space:]]+|[[:space:]]+$/, "", $2); print $2}' 2>/dev/null)
if [ -n "$CHAINLINK_IMAGE" ]; then
echo "CHAINLINK_IMAGE=$CHAINLINK_IMAGE" >> $GITHUB_ENV
else
echo "No Chainlink Image found in base64-ed config. Exiting"
exit 1
echo "No Chainlink Image found in base64-ed config"
fi
if [ -n "$CHAINLINK_VERSION" ]; then
echo "CHAINLINK_VERSION=$CHAINLINK_VERSION" >> $GITHUB_ENV
else
echo "No Chainlink Version found in base64-ed config. Exiting"
exit 1
fi
if [ -n "$NETWORKS" ]; then
echo "NETWORKS=$NETWORKS" >> $GITHUB_ENV
Expand Down
Loading

0 comments on commit 901609e

Please sign in to comment.