[TT-922, TT-943] Migrate automation and log poller tests to Seth (#12688

)

* WIP#1

* fix load test

* lints

* lints again

* uncomment tests

* fix logtrigger mercury test

* set privilege before starting consumer

* WIP#2

* WIP#2 - with high concurrency upkeep registration fails

* more or less working

* WIP#4

* small fixes

* fix lints

* use latest Seth

* Migrated keeper benchmark, kind of

* migrate keeper smoke tests

* bunch of optimisation

* migrate automation chaos test

* migrate log poller tests to Seth

* use anysynced key with log poller

* fix some of failing smoke tests

* fix go mod

* use correct Seth version

* better automation some test structure

* better automation some test structure

* go mod

* fix issues with missing LINK funds

* fix keeper test

* fix automation test

* clean up, comments, migrate keeper benchmark test

* mend

* use latest Seth

* cleanups, comments, etc

* clean ups, automation tests + keeper tests now always test multiple contracts

* use newer action version

* debug results when upkeeps fails

* run automation tests with post-failure debug

* run automation tests with post-failure debug

* better staleness report

* more debug for basic automation tests

* more assertions

* try with sleep

* increase performance check timeout to 3m

* separate config from consumer start

* more debug for performance chcek

* try with just 1 contract

* one more clean up

* run keeper tests in CI to test them

* solve automation load test registration failures

* subscribe to headers in benchmark test to observe upkeeps

* centralised ephemeral keys on testnet check, fixed keeper smoke tests, added root key buffer check

* do not fail, when on testnet and ephemeral keys are !=0, set them 0 instead

* don't fail in test config, display warning if eph addresses are set and set them to 0

* fix possible nil pointer, remove block range timeout, mutex for error appending

* update seth version

* use buffered channel to read subscription headers from EVM node

* benchmark test increase RR mem and set DEBUG log level

* use CTF's concurrent executor to deploy contracts in parallel

* change architecture so that we have only 1 subscription and then we fan out the header to all goroutines; use SafeEVMHeader

* benchmark test increase geth disk and gaslimit

* decrease log level, latest seth and ctf

* update CTF dependency

* use concurrent executor in automation load test

* remove local replace

* add header subscription reconnect handling

* print return funds error

* limit automation load test deployment concurrency

* latest Seth + fix gun

* do not run keeper smoke tests on each commit anymore

* add one more error message

* use tagged CTF

* use latest Seth

* use latest Seth

* fix go.mod

* forget about smart reconnect, just do it in a loop

* use latest Seth

* check if context has error before executing any of contract calling wrapper functions

* prettify code + latest Seth

* update Seth

* optional retrying of read-only contract calls; used only by keeper benchmark test

* update Seth, estimate gas for funds sending

* fix new automation test, update Seth to v1.0.7

* increase timeout

* fix max price smoke test

* fix CreateOCRKeeperJobs for v2.0

* remove resolved TODO

* code review changes

* code review changes

* code review changes vol 2

* use CTF release version

* bump mod version

---------

Co-authored-by: anirudhwarrier <[email protected]>

.github/actions/build-test-image/action.yml

@@ -34,7 +34,7 @@ runs:
     # Base Test Image Logic
     - name: Get CTF Version
       id: version
-      uses: smartcontractkit/chainlink-github-actions/chainlink-testing-framework/mod-version@5dd916d08c03cb5f9a97304f4f174820421bb946 # v2.3.11
+      uses: smartcontractkit/chainlink-github-actions/chainlink-testing-framework/mod-version@b6e5189ee90d0246c915da35c020c5822f233d42 # v2.3.13
       with:
         go-project-path: ./integration-tests
         module-name: github.com/smartcontractkit/chainlink-testing-framework

.github/workflows/automation-benchmark-tests.yml

@@ -48,7 +48,7 @@ jobs:
       - name: Parse base64 config
         uses: ./.github/actions/setup-parse-base64-config
         with:
-          base64Config: ${{ env.BASE64_CONFIG_OVERRIDE }}                 
+          base64Config: ${{ env.BASE64_CONFIG_OVERRIDE }}
       - name: Send details to Step Summary
         shell: bash
         run: |
@@ -57,7 +57,7 @@ jobs:
           echo "### chainlink-tests image tag for this test run :ship:" >>$GITHUB_STEP_SUMMARY
           echo "\`${GITHUB_SHA}\`" >>$GITHUB_STEP_SUMMARY
           echo "### Networks on which test was run" >>$GITHUB_STEP_SUMMARY
-          echo "\`${{ env.NETWORKS }}\`" >>$GITHUB_STEP_SUMMARY          
+          echo "\`${{ env.NETWORKS }}\`" >>$GITHUB_STEP_SUMMARY
       - name: Build Test Image
         uses: ./.github/actions/build-test-image
         with:
@@ -75,6 +75,8 @@ jobs:
           INTERNAL_DOCKER_REPO: ${{ secrets.QA_AWS_ACCOUNT_NUMBER }}.dkr.ecr.${{ secrets.QA_AWS_REGION }}.amazonaws.com
           TEST_TYPE: ${{ github.event.inputs.testType }}
           TEST_TEST_TYPE: ${{ github.event.inputs.testType }}
+          RR_MEM: 4Gi
+          TEST_LOG_LEVEL: info
         with:
           test_command_to_run: cd integration-tests && go test -timeout 30m -v -run ^TestAutomationBenchmark$ ./benchmark -count=1
           test_download_vendor_packages_command: make gomod

.github/workflows/integration-tests.yml

@@ -71,7 +71,7 @@ jobs:
           echo "should-enforce=$SHOULD_ENFORCE" >> $GITHUB_OUTPUT
       - name: Enforce CTF Version
         if: steps.condition-check.outputs.should-enforce == 'true'
-        uses: smartcontractkit/chainlink-github-actions/chainlink-testing-framework/mod-version@5dd916d08c03cb5f9a97304f4f174820421bb946 # v2.3.11
+        uses: smartcontractkit/chainlink-github-actions/chainlink-testing-framework/mod-version@e29366cdecfe6befff9ab8c3cfe4825218505d58 # v2.3.16
         with:
           go-project-path: ./integration-tests
           module-name: github.com/smartcontractkit/chainlink-testing-framework
@@ -468,7 +468,7 @@ jobs:
         with:
           name: cl-node-coverage-data-${{ matrix.product.name }}
           path: .covdata
-          retention-days: 1          
+          retention-days: 1
 
   eth-smoke-tests-matrix:
     if: ${{ !contains(join(github.event.pull_request.labels.*.name, ' '), 'skip-smoke-tests') }}
@@ -796,7 +796,7 @@ jobs:
   # Run the setup if the matrix finishes but this time save the cache if we have a cache hit miss
   # this will also only run if both of the matrix jobs pass
   eth-smoke-go-mod-cache:
-    
+
     environment: integration
     needs: [eth-smoke-tests]
     runs-on: ubuntu-latest
@@ -917,7 +917,7 @@ jobs:
         with:
           name: cl-node-coverage-data-migration-tests
           path: .covdata
-          retention-days: 1          
+          retention-days: 1
       - name: Notify Slack
         if: failure() && github.event_name != 'workflow_dispatch'
         uses: slackapi/slack-github-action@6c661ce58804a1a20f6dc5fbee7f0381b469e001 # v1.25.0
@@ -1210,12 +1210,12 @@ jobs:
           QA_KUBECONFIG: ""
           run_setup: false
           go_coverage_src_dir: /var/tmp/go-coverage
-          go_coverage_dest_dir: ${{ github.workspace }}/.covdata          
+          go_coverage_dest_dir: ${{ github.workspace }}/.covdata
 
       - name: Upload Coverage Data
         uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
         with:
           name: cl-node-coverage-data-solana-tests
           path: .covdata
-          retention-days: 1          
+          retention-days: 1
 

.gitignore

@@ -67,6 +67,11 @@ tmp-manifest-*.yaml
 ztarrepo.tar.gz
 **/test-ledger/*
 __debug_bin*
+.test_summary/
+.run.id
+integration-tests/**/traces/
+benchmark_report.csv
+benchmark_summary.json
 
 # goreleaser builds
 cosign.*
@@ -87,10 +92,6 @@ go.work*
 # This sometimes shows up for some reason
 tools/flakeytests/coverage.txt
 
-# Integration tests create these files
-.test_summary/
-.run.id
-
 # Fuzz tests can create these files
 **/testdata/fuzz/*
 

integration-tests/actions/automation_ocr_helpers.go

@@ -193,8 +193,10 @@ func CreateOCRKeeperJobs(
 		contractVersion = "v2.1+"
 	} else if registryVersion == ethereum.RegistryVersion_2_1 {
 		contractVersion = "v2.1"
+	} else if registryVersion == ethereum.RegistryVersion_2_0 {
+		contractVersion = "v2.0"
 	} else {
-		require.FailNow(t, "v2.0, v2.1, and v2.2 are the only supported versions")
+		require.FailNow(t, fmt.Sprintf("v2.0, v2.1, and v2.2 are the only supported versions, but got something else: %v (iota)", registryVersion))
 	}
 
 	bootstrapSpec := &client.OCR2TaskJobSpec{