TT-1831 Extend flaky test detector to run all core repository tests n…

…ightly (#15222)

* Move runWithRace to extraArgs

* Update flaky test worfklow to support running all tests

* TO REVERT: Remove most of deployment unit test

* update workflow

* fix

* fix

* fix

* debug

* fix

* change workflow name

* fix

* Update Github Summary

* TO REVERT: Remove one unit test

* Fix flakeguard binary path

* Fix

* Update runner counts

* Add nightly workflow for flaky test detector

* Fix

* Upload all test results as short lived artifact

* Fix total repeat count

* Revert "TO REVERT: Remove one unit test"

This reverts commit 5e0f921.

* Fix

* Add all_tests_runner arg

* Fix default all tests runner

* TO REVERT: fail test

* Update go project pretty name

* Fix

* Revert "TO REVERT: Remove most of deployment unit test"

This reverts commit 0c8418d.

* Update summary

* Fix

* Reapply "TO REVERT: Remove most of deployment unit test"

This reverts commit 7252d78.

* Update workflow names

* Do not print failed tests to console by default

* Update reporting

* Update job name

* Fix

* Bump flakeguard

* Fix workflow condition

* Remove debug step

* fix split-packages step

* fix 2

* Show pass ratio percentage

* Update nightly detector workflow

* Revert removing unit tests

* Fix ci-core.yml

* TEST: update unit test

* TEST: FAil deployment test

* Update slack notification

* TEST: fail test

* bump flakeguard

* Update name

* Revert "TEST: fail test"

This reverts commit 8c272b8.

* Revert "TEST: FAil deployment test"

This reverts commit e08d2b6.

* Fix test

* Bump flakeguard

* Use min pass ratio to not report tests as flaky if they failed in every run

* Change flaky threshold to 95%

* Bump flakeguard

* Print failed tests in github logs for PRs

* Run tests 3 times in PRs

* TEST: always fail test

* Bump flakeguard

* Fix

* Fix test

* add test that randomly passes or fails

* TO REVERT: repeat core tests 5 times

* Update min pass ratio and threshold

* Revert "add test that randomly passes or fails"

This reverts commit 43e7c0b.

* Remove test-results artifacts after 1 day

* Bump flakeguard and do not fail it when test fail but not flaky

* Fix

* TO REVERT: always fail test

* Update

* TO REVERT: add randomly failing test

* Revert "TO REVERT: add randomly failing test"

This reverts commit 5254cde.

* Fix test

* Bump

* Bump all_tests_runner_count for nightly

* Set min_pass_ratio=0 for nightly

* Set timeout for job

* Update

.github/workflows/ci-core.yml

@@ -463,20 +463,19 @@ jobs:
           SONAR_SCANNER_OPTS: "-Xms6g -Xmx8g"
 
   trigger-flaky-test-detection-for-root-project:
-    name: Find New Flaky Tests In Root Project
+    name: Find New Flaky Tests In Chainlink Project
     uses: ./.github/workflows/find-new-flaky-tests.yml
     if: ${{ github.event_name == 'pull_request' }}
     with:
       repoUrl: 'https://github.com/smartcontractkit/chainlink'
       projectPath: '.'
       baseRef: ${{ github.base_ref }}
       headRef: ${{ github.head_ref }}
-      runThreshold: '1'
-      runWithRace: true
+      runThreshold: '0.99'
       findByTestFilesDiff: true
       findByAffectedPackages: false
       slackNotificationAfterTestsChannelId: 'C07TRF65CNS' #flaky-test-detector-notifications
-      extraArgs: '{ "skipped_tests": "TestChainComponents" }'
+      extraArgs: '{ "skipped_tests": "TestChainComponents", "run_with_race": "true", "print_failed_tests": "true", "test_repeat_count": "3", "min_pass_ratio": "0.01" }'
     secrets:
       SLACK_BOT_TOKEN: ${{ secrets.QA_SLACK_API_KEY }}
 
@@ -490,12 +489,11 @@ jobs:
       projectPath: 'deployment'
       baseRef: ${{ github.base_ref }}
       headRef: ${{ github.head_ref }}
-      runThreshold: '1'
-      runWithRace: true
+      runThreshold: '0.99'
       findByTestFilesDiff: true
       findByAffectedPackages: false
       slackNotificationAfterTestsChannelId: 'C07TRF65CNS' #flaky-test-detector-notifications
-      extraArgs: '{ "skipped_tests": "TestAddLane" }'
+      extraArgs: '{ "skipped_tests": "TestAddLane", "run_with_race": "true", "print_failed_tests": "true", "test_repeat_count": "3", "min_pass_ratio": "0.01" }'
     secrets:
       SLACK_BOT_TOKEN: ${{ secrets.QA_SLACK_API_KEY }}