Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[TT-922, TT-943] Migrate automation and log poller tests to Seth #12688

Merged
merged 122 commits into from
May 13, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
122 commits
Select commit Hold shift + click to select a range
df237c8
WIP#1
Tofel Apr 3, 2024
14f6da5
Merge branch 'develop' into tt_922_automation_seth
Tofel Apr 4, 2024
a1096e0
fix load test
Tofel Apr 4, 2024
0eaed4a
lints
Tofel Apr 4, 2024
98f332b
lints again
Tofel Apr 4, 2024
bfae21c
Merge branch 'develop' into tt_922_automation_seth
Tofel Apr 4, 2024
69101e5
uncomment tests
Tofel Apr 4, 2024
a5e31cc
fix logtrigger mercury test
anirudhwarrier Apr 4, 2024
021f42f
set privilege before starting consumer
Tofel Apr 5, 2024
63c00e6
Merge branch 'tt_922_automation_seth-fix' into tt_922_automation_seth
Tofel Apr 5, 2024
b0228ad
WIP#2
Tofel Apr 8, 2024
e39db37
WIP#2 - with high concurrency upkeep registration fails
Tofel Apr 9, 2024
fd6f513
more or less working
Tofel Apr 10, 2024
f21fa49
WIP#4
Tofel Apr 10, 2024
3639ca7
small fixes
Tofel Apr 11, 2024
c8de136
Merge branch 'develop' into tt_922_automation_seth
Tofel Apr 11, 2024
6effd25
fix lints
Tofel Apr 11, 2024
63f4323
use latest Seth
Tofel Apr 11, 2024
914a4c2
Migrated keeper benchmark, kind of
Tofel Apr 11, 2024
12d6277
migrate keeper smoke tests
Tofel Apr 11, 2024
ce9d845
bunch of optimisation
Tofel Apr 12, 2024
00e17d8
migrate automation chaos test
Tofel Apr 12, 2024
d66942a
migrate log poller tests to Seth
Tofel Apr 15, 2024
5fabd1d
use anysynced key with log poller
Tofel Apr 15, 2024
f2bf001
fix some of failing smoke tests
Tofel Apr 15, 2024
24e2d3f
fix go mod
Tofel Apr 15, 2024
443b768
use correct Seth version
Tofel Apr 15, 2024
be469a8
better automation some test structure
Tofel Apr 15, 2024
6675182
better automation some test structure
Tofel Apr 15, 2024
6848b2e
Merge branch 'tt_943_log_poller_seth' of github.com:smartcontractkit/…
Tofel Apr 15, 2024
94d0090
go mod
Tofel Apr 15, 2024
0bdc6eb
fix issues with missing LINK funds
Tofel Apr 15, 2024
316c8c6
fix keeper test
Tofel Apr 15, 2024
cc92f2b
fix automation test
Tofel Apr 15, 2024
2f1e85f
clean up, comments, migrate keeper benchmark test
Tofel Apr 16, 2024
e50637c
mend
Tofel Apr 16, 2024
7a30cab
Merge branch 'develop' of github.com:smartcontractkit/chainlink into …
Tofel Apr 16, 2024
677fa9f
Merge branch 'develop' into tt_922_automation_seth
Tofel Apr 16, 2024
21aa2df
use latest Seth
Tofel Apr 16, 2024
5d26fbb
cleanups, comments, etc
Tofel Apr 17, 2024
9139356
Merge branch 'develop' of github.com:smartcontractkit/chainlink into …
Tofel Apr 17, 2024
1dd5e07
clean ups, automation tests + keeper tests now always test multiple c…
Tofel Apr 17, 2024
b2fce41
use newer action version
Tofel Apr 17, 2024
4ef09ac
debug results when upkeeps fails
Tofel Apr 17, 2024
611e031
Merge branch 'develop' of github.com:smartcontractkit/chainlink into …
Tofel Apr 17, 2024
de09259
Merge branch 'develop' into tt_922_automation_seth
Tofel Apr 17, 2024
6c162ce
run automation tests with post-failure debug
Tofel Apr 17, 2024
f63f56b
run automation tests with post-failure debug
Tofel Apr 17, 2024
570a139
Merge branch 'develop' of github.com:smartcontractkit/chainlink into …
Tofel Apr 17, 2024
0a72069
Merge branch 'develop' into tt_922_automation_seth
Tofel Apr 17, 2024
2373525
Merge branch 'tt_922_automation_seth' of github.com:smartcontractkit/…
Tofel Apr 17, 2024
34de8d0
better staleness report
Tofel Apr 17, 2024
0b393ca
more debug for basic automation tests
Tofel Apr 17, 2024
390a142
more assertions
Tofel Apr 17, 2024
17d8cf1
try with sleep
Tofel Apr 17, 2024
1d31500
increase performance check timeout to 3m
Tofel Apr 17, 2024
be653ae
separate config from consumer start
Tofel Apr 17, 2024
86c055d
more debug for performance chcek
Tofel Apr 17, 2024
021c6d3
try with just 1 contract
Tofel Apr 18, 2024
44ce6ff
one more clean up
Tofel Apr 18, 2024
1bb40d9
run keeper tests in CI to test them
Tofel Apr 18, 2024
e429fb9
solve automation load test registration failures
Tofel Apr 18, 2024
8e556bd
Merge branch 'develop' of github.com:smartcontractkit/chainlink into …
Tofel Apr 19, 2024
7e77724
subscribe to headers in benchmark test to observe upkeeps
Tofel Apr 19, 2024
a0d77c1
Merge branch 'develop' of github.com:smartcontractkit/chainlink into …
Tofel Apr 19, 2024
b04528a
centralised ephemeral keys on testnet check, fixed keeper smoke tests…
Tofel Apr 22, 2024
eb17edf
Merge branch 'develop' of github.com:smartcontractkit/chainlink into …
Tofel Apr 22, 2024
5fd4d3d
Merge branch 'develop' into tt_922_automation_seth
Tofel Apr 22, 2024
99dd165
Merge branch 'develop' of github.com:smartcontractkit/chainlink into …
Tofel Apr 24, 2024
da267d8
do not fail, when on testnet and ephemeral keys are !=0, set them 0 i…
Tofel Apr 24, 2024
c2a6f5c
Merge branch 'develop' into tt_922_automation_seth
Tofel Apr 24, 2024
2b9bf7c
don't fail in test config, display warning if eph addresses are set a…
Tofel Apr 24, 2024
0223e5a
fix possible nil pointer, remove block range timeout, mutex for error…
Tofel Apr 24, 2024
796a138
update seth version
Tofel Apr 24, 2024
67ae4c8
use buffered channel to read subscription headers from EVM node
Tofel Apr 25, 2024
ed8d37f
benchmark test increase RR mem and set DEBUG log level
anirudhwarrier Apr 25, 2024
c3a5942
use CTF's concurrent executor to deploy contracts in parallel
Tofel Apr 25, 2024
e3de908
Merge pull request #12965 from smartcontractkit/tt_922_automation_set…
Tofel Apr 25, 2024
3a4979a
change architecture so that we have only 1 subscription and then we f…
Tofel Apr 25, 2024
74bc12d
benchmark test increase geth disk and gaslimit
anirudhwarrier Apr 26, 2024
399dd98
decrease log level, latest seth and ctf
Tofel Apr 26, 2024
5d97ff6
update CTF dependency
Tofel Apr 26, 2024
472c537
Merge branch 'tt_922_automation_seth' of github.com:smartcontractkit/…
Tofel Apr 26, 2024
c5a68a6
use concurrent executor in automation load test
Tofel Apr 26, 2024
fa7ed30
Merge branch 'develop' of github.com:smartcontractkit/chainlink into …
Tofel Apr 26, 2024
817915b
remove local replace
Tofel Apr 26, 2024
0d54262
add header subscription reconnect handling
Tofel Apr 29, 2024
22fe929
Merge branch 'develop' of github.com:smartcontractkit/chainlink into …
Tofel Apr 29, 2024
09867b1
Merge branch 'develop' into tt_922_automation_seth
Tofel Apr 29, 2024
d4cfdd6
print return funds error
Tofel Apr 29, 2024
3baa7ad
limit automation load test deployment concurrency
Tofel Apr 29, 2024
b36bdfd
latest Seth + fix gun
Tofel Apr 29, 2024
8b55cec
Merge branch 'develop' of github.com:smartcontractkit/chainlink into …
Tofel Apr 29, 2024
2abc2cd
Merge branch 'develop' into tt_922_automation_seth
Tofel Apr 29, 2024
76a113c
do not run keeper smoke tests on each commit anymore
Tofel Apr 29, 2024
e208572
add one more error message
Tofel Apr 29, 2024
939eab9
use tagged CTF
Tofel Apr 29, 2024
c304997
use latest Seth
Tofel Apr 11, 2024
f7edc0f
Merge branch 'develop' into tt_922_automation_seth
Tofel Apr 30, 2024
69fac66
use latest Seth
Tofel Apr 30, 2024
47e164e
fix go.mod
Tofel Apr 30, 2024
1b9228b
forget about smart reconnect, just do it in a loop
Tofel Apr 30, 2024
90a9515
use latest Seth
Tofel Apr 11, 2024
40e1dbf
Merge branch 'develop' into tt_922_automation_seth
Tofel May 6, 2024
f897175
check if context has error before executing any of contract calling w…
Tofel May 6, 2024
55abcc9
prettify code + latest Seth
Tofel May 6, 2024
8068b9f
update Seth
Tofel May 6, 2024
86f638a
optional retrying of read-only contract calls; used only by keeper be…
Tofel May 7, 2024
daaf9d3
Merge branch 'develop' into tt_922_automation_seth
Tofel May 7, 2024
897dd96
update Seth, estimate gas for funds sending
Tofel May 7, 2024
a7ed0fb
fix new automation test, update Seth to v1.0.7
Tofel May 7, 2024
90257db
increase timeout
Tofel May 7, 2024
461bc74
fix max price smoke test
Tofel May 8, 2024
78e9ab0
Merge branch 'develop' into tt_922_automation_seth
Tofel May 8, 2024
5a3b7e8
fix CreateOCRKeeperJobs for v2.0
anirudhwarrier May 8, 2024
37a6d36
remove resolved TODO
anirudhwarrier May 8, 2024
edc4a24
code review changes
Tofel May 8, 2024
0bd7419
Merge branch 'develop' into tt_922_automation_seth
Tofel May 9, 2024
37f6776
code review changes
Tofel May 10, 2024
e5c3bd7
code review changes vol 2
Tofel May 10, 2024
9ea9aea
use CTF release version
Tofel May 10, 2024
6591af7
bump mod version
Tofel May 10, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/actions/build-test-image/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ runs:
# Base Test Image Logic
- name: Get CTF Version
id: version
uses: smartcontractkit/chainlink-github-actions/chainlink-testing-framework/mod-version@5dd916d08c03cb5f9a97304f4f174820421bb946 # v2.3.11
uses: smartcontractkit/chainlink-github-actions/chainlink-testing-framework/mod-version@b6e5189ee90d0246c915da35c020c5822f233d42 # v2.3.13
with:
go-project-path: ./integration-tests
module-name: github.com/smartcontractkit/chainlink-testing-framework
Expand Down
6 changes: 4 additions & 2 deletions .github/workflows/automation-benchmark-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ jobs:
- name: Parse base64 config
uses: ./.github/actions/setup-parse-base64-config
with:
base64Config: ${{ env.BASE64_CONFIG_OVERRIDE }}
base64Config: ${{ env.BASE64_CONFIG_OVERRIDE }}
- name: Send details to Step Summary
shell: bash
run: |
Expand All @@ -57,7 +57,7 @@ jobs:
echo "### chainlink-tests image tag for this test run :ship:" >>$GITHUB_STEP_SUMMARY
echo "\`${GITHUB_SHA}\`" >>$GITHUB_STEP_SUMMARY
echo "### Networks on which test was run" >>$GITHUB_STEP_SUMMARY
echo "\`${{ env.NETWORKS }}\`" >>$GITHUB_STEP_SUMMARY
echo "\`${{ env.NETWORKS }}\`" >>$GITHUB_STEP_SUMMARY
- name: Build Test Image
uses: ./.github/actions/build-test-image
with:
Expand All @@ -75,6 +75,8 @@ jobs:
INTERNAL_DOCKER_REPO: ${{ secrets.QA_AWS_ACCOUNT_NUMBER }}.dkr.ecr.${{ secrets.QA_AWS_REGION }}.amazonaws.com
TEST_TYPE: ${{ github.event.inputs.testType }}
TEST_TEST_TYPE: ${{ github.event.inputs.testType }}
RR_MEM: 4Gi
TEST_LOG_LEVEL: info
with:
test_command_to_run: cd integration-tests && go test -timeout 30m -v -run ^TestAutomationBenchmark$ ./benchmark -count=1
test_download_vendor_packages_command: make gomod
Expand Down
12 changes: 6 additions & 6 deletions .github/workflows/integration-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -71,7 +71,7 @@ jobs:
echo "should-enforce=$SHOULD_ENFORCE" >> $GITHUB_OUTPUT
- name: Enforce CTF Version
if: steps.condition-check.outputs.should-enforce == 'true'
uses: smartcontractkit/chainlink-github-actions/chainlink-testing-framework/mod-version@5dd916d08c03cb5f9a97304f4f174820421bb946 # v2.3.11
uses: smartcontractkit/chainlink-github-actions/chainlink-testing-framework/mod-version@e29366cdecfe6befff9ab8c3cfe4825218505d58 # v2.3.16
with:
go-project-path: ./integration-tests
module-name: github.com/smartcontractkit/chainlink-testing-framework
Expand Down Expand Up @@ -468,7 +468,7 @@ jobs:
with:
name: cl-node-coverage-data-${{ matrix.product.name }}
path: .covdata
retention-days: 1
retention-days: 1

eth-smoke-tests-matrix:
if: ${{ !contains(join(github.event.pull_request.labels.*.name, ' '), 'skip-smoke-tests') }}
Expand Down Expand Up @@ -796,7 +796,7 @@ jobs:
# Run the setup if the matrix finishes but this time save the cache if we have a cache hit miss
# this will also only run if both of the matrix jobs pass
eth-smoke-go-mod-cache:

environment: integration
needs: [eth-smoke-tests]
runs-on: ubuntu-latest
Expand Down Expand Up @@ -917,7 +917,7 @@ jobs:
with:
name: cl-node-coverage-data-migration-tests
path: .covdata
retention-days: 1
retention-days: 1
- name: Notify Slack
if: failure() && github.event_name != 'workflow_dispatch'
uses: slackapi/slack-github-action@6c661ce58804a1a20f6dc5fbee7f0381b469e001 # v1.25.0
Expand Down Expand Up @@ -1215,12 +1215,12 @@ jobs:
QA_KUBECONFIG: ""
run_setup: false
go_coverage_src_dir: /var/tmp/go-coverage
go_coverage_dest_dir: ${{ github.workspace }}/.covdata
go_coverage_dest_dir: ${{ github.workspace }}/.covdata

- name: Upload Coverage Data
uses: actions/upload-artifact@65462800fd760344b1a7b4382951275a0abb4808 # v4.3.3
with:
name: cl-node-coverage-data-solana-tests
path: .covdata
retention-days: 1
retention-days: 1

9 changes: 5 additions & 4 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,11 @@ tmp-manifest-*.yaml
ztarrepo.tar.gz
**/test-ledger/*
__debug_bin*
.test_summary/
.run.id
integration-tests/**/traces/
benchmark_report.csv
benchmark_summary.json

# goreleaser builds
cosign.*
Expand All @@ -87,10 +92,6 @@ go.work*
# This sometimes shows up for some reason
tools/flakeytests/coverage.txt

# Integration tests create these files
.test_summary/
.run.id

# Fuzz tests can create these files
**/testdata/fuzz/*

Expand Down
4 changes: 3 additions & 1 deletion integration-tests/actions/automation_ocr_helpers.go
Original file line number Diff line number Diff line change
Expand Up @@ -193,8 +193,10 @@ func CreateOCRKeeperJobs(
contractVersion = "v2.1+"
} else if registryVersion == ethereum.RegistryVersion_2_1 {
contractVersion = "v2.1"
} else if registryVersion == ethereum.RegistryVersion_2_0 {
contractVersion = "v2.0"
} else {
require.FailNow(t, "v2.0, v2.1, and v2.2 are the only supported versions")
require.FailNow(t, fmt.Sprintf("v2.0, v2.1, and v2.2 are the only supported versions, but got something else: %v (iota)", registryVersion))
}

bootstrapSpec := &client.OCR2TaskJobSpec{
Expand Down
Loading
Loading