Create reusable workflow for E2E tests

[lukaszcl]

This PR introduces a Reusable GitHub Workflow for E2E tests, centralizing all the necessary logic to run various types of tests—such as Docker and K8s remote runner—within a single workflow. This approach allows other E2E test workflows to utilize this central workflow at different stages of the software development process, whether it's for every commit, within the merge queue, nightly, or on-demand.

The workflow sources the tests to run from the .github/e2e-tests.yml file, where each test or group of tests is defined with specific parameters: test_cmd to dictate how to execute the test, runs_on to specify the required GitHub runner, and a workflows parameter that outlines when each test should be executed (e.g., on every pull request or nightly).

  - id: integration-tests/soak/ocr_test.go:^TestOCRv2Soak$
    path: integration-tests/soak/ocr_test.go
    test_env_type: k8s-remote-runner
    runs_on: ubuntu-latest
    test_cmd: cd integration-tests/ && go test soak/ocr_test.go -v -test.run ^TestOCRv2Soak$ -test.parallel=1 -timeout 30m -count=1
    test_config_override_required: true
    test_secrets_required: true
    test_inputs:
      test_suite: soak
      
  # Example of 1 runner for all tests in integration-tests/smoke/ocr_test.go
  - id: integration-tests/smoke/ocr_test.go:* 
    path: integration-tests/smoke/ocr_test.go
    test_env_type: docker
    runs_on: ubuntu-latest
    workflows:
      - Run PR E2E Tests
      - Run Nightly E2E Tests
    test_cmd: cd integration-tests/ && go test smoke/ocr_test.go -timeout 30m -count=1 -test.parallel=2 -json
    pyroscope_env: ci-smoke-ocr-evm-simulated      

The run-e2e-tests-reusable-workflow.yml workflow can be utilized in various ways:

• Specify Test IDs: Provide comma-separated test IDs in the test_ids input to run specific tests. For example, see the run-selected-e2e-tests.yml workflow.
• Run Tests by Workflow: Use the test_workflow input to run all tests associated with a particular workflow. For example, see the run-nightly-e2e-tests.yml workflow.
• Custom Test List: Input a list of tests with custom settings using the test_list input. For example, see the run-automation-ondemand-e2e-tests.yml workflow.

To learn more about new CI Workflows for E2E Tests, please visit https://smartcontract-it.atlassian.net/wiki/spaces/TT/pages/815497220/CI+Workflows+for+E2E+Tests

Note: This update does not remove existing workflows for now. Changes to integrate workflows with the E2E tests reusable workflow will be addressed in separate PRs. More details are provided below.

---

Full scope of this PR:

1. Add reusable workflow to run E2E tests: run-e2e-tests-reusable-workflow.yml
2. Add a workflow to run any E2E test from .github/e2e-tests.yml on demand: run-selected-e2e-tests.yml (example build)

   gh workflow run run-selected-e2e-tests.yml --ref TT-1297-Create-nightly-pipeline-for-all-e2e-tests-in-core -f chainlink_version=develop -f test_ids="integration-tests/migration/upgrade_version_test.go:*"
   

3. Add a workflow to run tests nightly: run-nightly-e2e-tests.yml. Tests to run nightly have "Run Nightly E2E Tests" in workflows field in .github/e2e-tests.yml file. Example build.
4. Add run-automation-ondemand-e2e-tests.yml that works the same as automation-ondemand-tests.yml but uses the reusable workflow (example build). Once it's tested we can remove the old workflow. run-automation-ondemand-e2e-tests.yml is an example how to run tests with custom test inputs
5. Add go tool to detect dead-tests (tests that are not part of any CI workflow). Learn more here. This is disabled by default for now
6. Add go tool to create default test config override (./.github/actions/create-default-e2e-config-override) and use it instead of the old bash script
7. Add jobs to reusable workflow to easily build required Chainlink Node image and Chainlink Plugins image. Required versions can be set using require_chainlink_image_versions_in_qa_ecr and require_chainlink_plugin_versions_in_qa_ecr workflow inputs
8. Add default Slack notification for the E2E tests reusable workflow

Not in the scope:

1. Refactoring existing workflows to utilise the new Reusable E2E Tests Workflow will be addressed in separate PRs
2. Allow to pass full test_config_base64_override as workflow input. This will be enabled once TestConfig does not have any secrets. Related ticket https://smartcontract-it.atlassian.net/browse/TT-1392

[cl-sonarqube-production]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarQube