[TT-922, TT-943] Migrate automation and log poller tests to Seth #12688
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Tofel
changed the title
kalverra
previously approved these changes
May 10, 2024
anirudhwarrier
previously approved these changes
May 10, 2024
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
kalverra
approved these changes
May 10, 2024
tateexon
approved these changes
May 10, 2024
|
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
github-merge-queue
bot
removed this pull request from the merge queue due to failed status checks
Tofel
added a commit
that referenced
this pull request
May 13, 2024
) * WIP#1 * fix load test * lints * lints again * uncomment tests * fix logtrigger mercury test * set privilege before starting consumer * WIP#2 * WIP#2 - with high concurrency upkeep registration fails * more or less working * WIP#4 * small fixes * fix lints * use latest Seth * Migrated keeper benchmark, kind of * migrate keeper smoke tests * bunch of optimisation * migrate automation chaos test * migrate log poller tests to Seth * use anysynced key with log poller * fix some of failing smoke tests * fix go mod * use correct Seth version * better automation some test structure * better automation some test structure * go mod * fix issues with missing LINK funds * fix keeper test * fix automation test * clean up, comments, migrate keeper benchmark test * mend * use latest Seth * cleanups, comments, etc * clean ups, automation tests + keeper tests now always test multiple contracts * use newer action version * debug results when upkeeps fails * run automation tests with post-failure debug * run automation tests with post-failure debug * better staleness report * more debug for basic automation tests * more assertions * try with sleep * increase performance check timeout to 3m * separate config from consumer start * more debug for performance chcek * try with just 1 contract * one more clean up * run keeper tests in CI to test them * solve automation load test registration failures * subscribe to headers in benchmark test to observe upkeeps * centralised ephemeral keys on testnet check, fixed keeper smoke tests, added root key buffer check * do not fail, when on testnet and ephemeral keys are !=0, set them 0 instead * don't fail in test config, display warning if eph addresses are set and set them to 0 * fix possible nil pointer, remove block range timeout, mutex for error appending * update seth version * use buffered channel to read subscription headers from EVM node * benchmark test increase RR mem and set DEBUG log level * use CTF's concurrent executor to deploy contracts in parallel * change architecture so that we have only 1 subscription and then we fan out the header to all goroutines; use SafeEVMHeader * benchmark test increase geth disk and gaslimit * decrease log level, latest seth and ctf * update CTF dependency * use concurrent executor in automation load test * remove local replace * add header subscription reconnect handling * print return funds error * limit automation load test deployment concurrency * latest Seth + fix gun * do not run keeper smoke tests on each commit anymore * add one more error message * use tagged CTF * use latest Seth * use latest Seth * fix go.mod * forget about smart reconnect, just do it in a loop * use latest Seth * check if context has error before executing any of contract calling wrapper functions * prettify code + latest Seth * update Seth * optional retrying of read-only contract calls; used only by keeper benchmark test * update Seth, estimate gas for funds sending * fix new automation test, update Seth to v1.0.7 * increase timeout * fix max price smoke test * fix CreateOCRKeeperJobs for v2.0 * remove resolved TODO * code review changes * code review changes * code review changes vol 2 * use CTF release version * bump mod version --------- Co-authored-by: anirudhwarrier <[email protected]>
Tofel
added a commit
that referenced
this pull request
May 13, 2024
) * WIP#1 * fix load test * lints * lints again * uncomment tests * fix logtrigger mercury test * set privilege before starting consumer * WIP#2 * WIP#2 - with high concurrency upkeep registration fails * more or less working * WIP#4 * small fixes * fix lints * use latest Seth * Migrated keeper benchmark, kind of * migrate keeper smoke tests * bunch of optimisation * migrate automation chaos test * migrate log poller tests to Seth * use anysynced key with log poller * fix some of failing smoke tests * fix go mod * use correct Seth version * better automation some test structure * better automation some test structure * go mod * fix issues with missing LINK funds * fix keeper test * fix automation test * clean up, comments, migrate keeper benchmark test * mend * use latest Seth * cleanups, comments, etc * clean ups, automation tests + keeper tests now always test multiple contracts * use newer action version * debug results when upkeeps fails * run automation tests with post-failure debug * run automation tests with post-failure debug * better staleness report * more debug for basic automation tests * more assertions * try with sleep * increase performance check timeout to 3m * separate config from consumer start * more debug for performance chcek * try with just 1 contract * one more clean up * run keeper tests in CI to test them * solve automation load test registration failures * subscribe to headers in benchmark test to observe upkeeps * centralised ephemeral keys on testnet check, fixed keeper smoke tests, added root key buffer check * do not fail, when on testnet and ephemeral keys are !=0, set them 0 instead * don't fail in test config, display warning if eph addresses are set and set them to 0 * fix possible nil pointer, remove block range timeout, mutex for error appending * update seth version * use buffered channel to read subscription headers from EVM node * benchmark test increase RR mem and set DEBUG log level * use CTF's concurrent executor to deploy contracts in parallel * change architecture so that we have only 1 subscription and then we fan out the header to all goroutines; use SafeEVMHeader * benchmark test increase geth disk and gaslimit * decrease log level, latest seth and ctf * update CTF dependency * use concurrent executor in automation load test * remove local replace * add header subscription reconnect handling * print return funds error * limit automation load test deployment concurrency * latest Seth + fix gun * do not run keeper smoke tests on each commit anymore * add one more error message * use tagged CTF * use latest Seth * use latest Seth * fix go.mod * forget about smart reconnect, just do it in a loop * use latest Seth * check if context has error before executing any of contract calling wrapper functions * prettify code + latest Seth * update Seth * optional retrying of read-only contract calls; used only by keeper benchmark test * update Seth, estimate gas for funds sending * fix new automation test, update Seth to v1.0.7 * increase timeout * fix max price smoke test * fix CreateOCRKeeperJobs for v2.0 * remove resolved TODO * code review changes * code review changes * code review changes vol 2 * use CTF release version * bump mod version --------- Co-authored-by: anirudhwarrier <[email protected]>
github-merge-queue bot
pushed a commit
that referenced
this pull request
May 15, 2024
* add back functionality that checks cl node logs after tests for errors * add comment, newer CTF version * use CTF release version * ignore expected errors in cl node logs * fix go.mod * fix load tests * fix lints * fix lints again * fix another vrfv2 log error * use CTF release version * [TT-922, TT-943] Migrate automation and log poller tests to Seth (#12688) * WIP#1 * fix load test * lints * lints again * uncomment tests * fix logtrigger mercury test * set privilege before starting consumer * WIP#2 * WIP#2 - with high concurrency upkeep registration fails * more or less working * WIP#4 * small fixes * fix lints * use latest Seth * Migrated keeper benchmark, kind of * migrate keeper smoke tests * bunch of optimisation * migrate automation chaos test * migrate log poller tests to Seth * use anysynced key with log poller * fix some of failing smoke tests * fix go mod * use correct Seth version * better automation some test structure * better automation some test structure * go mod * fix issues with missing LINK funds * fix keeper test * fix automation test * clean up, comments, migrate keeper benchmark test * mend * use latest Seth * cleanups, comments, etc * clean ups, automation tests + keeper tests now always test multiple contracts * use newer action version * debug results when upkeeps fails * run automation tests with post-failure debug * run automation tests with post-failure debug * better staleness report * more debug for basic automation tests * more assertions * try with sleep * increase performance check timeout to 3m * separate config from consumer start * more debug for performance chcek * try with just 1 contract * one more clean up * run keeper tests in CI to test them * solve automation load test registration failures * subscribe to headers in benchmark test to observe upkeeps * centralised ephemeral keys on testnet check, fixed keeper smoke tests, added root key buffer check * do not fail, when on testnet and ephemeral keys are !=0, set them 0 instead * don't fail in test config, display warning if eph addresses are set and set them to 0 * fix possible nil pointer, remove block range timeout, mutex for error appending * update seth version * use buffered channel to read subscription headers from EVM node * benchmark test increase RR mem and set DEBUG log level * use CTF's concurrent executor to deploy contracts in parallel * change architecture so that we have only 1 subscription and then we fan out the header to all goroutines; use SafeEVMHeader * benchmark test increase geth disk and gaslimit * decrease log level, latest seth and ctf * update CTF dependency * use concurrent executor in automation load test * remove local replace * add header subscription reconnect handling * print return funds error * limit automation load test deployment concurrency * latest Seth + fix gun * do not run keeper smoke tests on each commit anymore * add one more error message * use tagged CTF * use latest Seth * use latest Seth * fix go.mod * forget about smart reconnect, just do it in a loop * use latest Seth * check if context has error before executing any of contract calling wrapper functions * prettify code + latest Seth * update Seth * optional retrying of read-only contract calls; used only by keeper benchmark test * update Seth, estimate gas for funds sending * fix new automation test, update Seth to v1.0.7 * increase timeout * fix max price smoke test * fix CreateOCRKeeperJobs for v2.0 * remove resolved TODO * code review changes * code review changes * code review changes vol 2 * use CTF release version * bump mod version --------- Co-authored-by: anirudhwarrier <[email protected]> * add exception to other chaos log poller smoke tests --------- Co-authored-by: anirudhwarrier <[email protected]>
github-merge-queue bot
pushed a commit
that referenced
this pull request
May 20, 2024
* use latest Seth * move test config to CTF * use latest CTF that fixes eth2 genesis generation * use latest Seth * use tagged CTF version * WIP#1 * WIP#2 * add automation-specifc node configuration to TOML * add node TOML config for OCR forwarders smoke tests * add CL node TOML configuration for keeper tests * add CL node TOML config for log poller * add OCR1/2 TOML config * add VRF* CL node TOML config * remove functional options for setting CL node config in tests, use TOML instead * go mod tidy * fix go mod * fix test env builder * increase node funding in runlog test * increase vrf node funding * add on-demand keeper smoke tests workflow * run keeper smoke tests once * fix int tests yaml? * [TT-922, TT-943] Migrate automation and log poller tests to Seth (#12688) * WIP#1 * fix load test * lints * lints again * uncomment tests * fix logtrigger mercury test * set privilege before starting consumer * WIP#2 * WIP#2 - with high concurrency upkeep registration fails * more or less working * WIP#4 * small fixes * fix lints * use latest Seth * Migrated keeper benchmark, kind of * migrate keeper smoke tests * bunch of optimisation * migrate automation chaos test * migrate log poller tests to Seth * use anysynced key with log poller * fix some of failing smoke tests * fix go mod * use correct Seth version * better automation some test structure * better automation some test structure * go mod * fix issues with missing LINK funds * fix keeper test * fix automation test * clean up, comments, migrate keeper benchmark test * mend * use latest Seth * cleanups, comments, etc * clean ups, automation tests + keeper tests now always test multiple contracts * use newer action version * debug results when upkeeps fails * run automation tests with post-failure debug * run automation tests with post-failure debug * better staleness report * more debug for basic automation tests * more assertions * try with sleep * increase performance check timeout to 3m * separate config from consumer start * more debug for performance chcek * try with just 1 contract * one more clean up * run keeper tests in CI to test them * solve automation load test registration failures * subscribe to headers in benchmark test to observe upkeeps * centralised ephemeral keys on testnet check, fixed keeper smoke tests, added root key buffer check * do not fail, when on testnet and ephemeral keys are !=0, set them 0 instead * don't fail in test config, display warning if eph addresses are set and set them to 0 * fix possible nil pointer, remove block range timeout, mutex for error appending * update seth version * use buffered channel to read subscription headers from EVM node * benchmark test increase RR mem and set DEBUG log level * use CTF's concurrent executor to deploy contracts in parallel * change architecture so that we have only 1 subscription and then we fan out the header to all goroutines; use SafeEVMHeader * benchmark test increase geth disk and gaslimit * decrease log level, latest seth and ctf * update CTF dependency * use concurrent executor in automation load test * remove local replace * add header subscription reconnect handling * print return funds error * limit automation load test deployment concurrency * latest Seth + fix gun * do not run keeper smoke tests on each commit anymore * add one more error message * use tagged CTF * use latest Seth * use latest Seth * fix go.mod * forget about smart reconnect, just do it in a loop * use latest Seth * check if context has error before executing any of contract calling wrapper functions * prettify code + latest Seth * update Seth * optional retrying of read-only contract calls; used only by keeper benchmark test * update Seth, estimate gas for funds sending * fix new automation test, update Seth to v1.0.7 * increase timeout * fix max price smoke test * fix CreateOCRKeeperJobs for v2.0 * remove resolved TODO * code review changes * code review changes * code review changes vol 2 * use CTF release version * bump mod version --------- Co-authored-by: anirudhwarrier <[email protected]> * fix issue introduced by merge conflict * adjust keeper & automation TOML configs * restore old integration-tests workflow that doesn't run keeper tests * fix double log stream shutdown * add back clNodeOpts * use tagged CTF version --------- Co-authored-by: Ilja Pavlovs <[email protected]> Co-authored-by: anirudhwarrier <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Migrated:
Excluded from ever being migrated:
Important details:
NewAutomationTestDocker(), which is used only by automation load test; instead Seth-based actions were created and used in migrated testsEVMClientwere renamed toLegacy*