Merge branch 'develop' into BCI-2644-refactor-evm-orms

integration-tests/README.md

@@ -14,7 +14,26 @@ If you have previously run these smoke tests using GitHub Actions or some sort o
 
 ## Configure
 
-See the [example.env](./example.env) file for environment variables you can set to configure things like network settings, Chainlink version, and log level. Remember to use `source .env` to activate your settings.
+We have finished first pass at moving the configuration from env vars to TOML files. Currently all product-related configuration is already in TOML files, but env vars still are used to control things like log level, Slack notifications and Kubernetes-related settings. See the [example.env](./example.env) file for environment variables.
+
+We have added what we think are sensible defaults for all products, you can find them in `./testconfig/<product>/<product>.toml` files. Each product folder contains also an `example.toml` file with all possible TOML keys and some description. Detailed description of TOML configuration can be found in [README.md](./testconfig/README.md), but if you want to run some tests using default value all you need to do is provide Chainlink image and version:
+```toml
+# ./testconfig/overrides.toml
+
+[ChainlinkImage]
+image = "your image name"
+version = "your tag"
+```
+
+You could also think about that config this way:
+```toml
+# ./testconfig/overrides.toml
+[ChainlinkImage]
+image = "${CHAINLINK_IMAGE}"
+version = "${CHAINLINK_VERSION}"
+```
+
+Of course above just and example, in real world no substitution will take place unless you use some templating tool, but it should give you an idea on how to move from env vars to TOML files. **Remember** your runtime configuration needs to be placed in `./testconfig/overrides.toml` file **that should never be committed**.
 
 ## Build
 
@@ -26,10 +45,10 @@ e.g.
 
 `make build_docker_image image=chainlink tag=test-tag`
 
-You'll want to set the `CHAINLINK_IMAGE` and `CHAINLINK_VERSION` env values appropriately as well. See [example.env](./example.env) for more details.
-
 ## Run
 
+Make sure you have `./testconfig/overrides.toml` file with your Chainlink image and version.
+
 `go test ./smoke/<product>_test.go`
 
 Most test files have a couple of tests, it's recommended to look into the file and focus on a specific one if possible. 90% of the time this will probably be the `Basic` test. See [ocr_test.go](./smoke/ocr_test.go) for example, which contains the `TestOCRBasic` test.

integration-tests/testconfig/README.md

@@ -0,0 +1,168 @@
+# TOML is the Ultimate Choice!
+
+## Introduction
+
+Final implementation has undergone minor adjustments in comparison to the approach by Adam Hamric, Anindita Ghosh, and Sergey Kudasov stated in the ADR. The primary changes are as follows:
+* `TEST_LOG_LEVEL` remains an environment variable, pending the release of version 2.
+* `TEST_TYPE` is also kept as an environment variable to facilitate dynamic configuration selection by some tests.
+* TOML configuration of Chainlink nodes themselves has not been added, awaiting version 2.
+* The hierarchy of configuration overrides has been streamlined for simplicity.
+
+By design, all test configurations are intended to reside within the `testconfig` package, organized into application-specific folders. However, the system can locate these configurations in any folder within the `integration-tests` directory, selecting the first one found. To identify the configurations in use, execute tests with the `debug` log level.
+
+The `testconfig` package serves as a centralized resource for accessing configurations across all products, including shared settings like logging and network preferences, as well as initial funding for Chainlink nodes. Product configurations, if present, are subjected to validation based on logical assumptions and observed code values. The `TestConfig` structure includes a `Save()` method, allowing for the preservation of test configurations after all adjustments have been applied.
+
+## Configuration and Overrides
+
+The order of precedence for overrides is as follows:
+* Environment variable `BASE64_CONFIG_OVERRIDE`
+* File `overrides.toml`
+* Product-specific file, e.g., `[product_name].toml`
+* The `default.toml` file
+
+The `BASE64_CONFIG_OVERRIDE` environment variable is primarily intended for use in continuous integration environments, enabling the substitution of default settings with confidential or user-specific parameters. For instance:
+
+```bash
+cat << EOF > config.toml
+[Network]
+selected_networks=["$SELECTED_NETWORKS"]
+
+[ChainlinkImage]
+image="$CHAINLINK_IMAGE"
+version="$CHAINLINK_VERSION"
+postgres_version="$CHAINLINK_POSTGRES_VERSION"
+
+[Pyroscope]
+enabled=$pyroscope_enabled
+server_url="$PYROSCOPE_SERVER"
+environment="$PYROSCOPE_ENVIRONMENT"
+key_secret="$PYROSCOPE_KEY"
+
+[Logging]
+test_log_collect=false
+run_id="$RUN_ID"
+
+[Logging.LogStream]
+log_targets=["$LOG_TARGETS"]
+
+[Logging.Loki]
+tenant_id="$LOKI_TENANT_ID"
+endpoint="$LOKI_ENDPOINT"
+basic_auth_secret="$LOKI_BASIC_AUTH"
+
+[Logging.Grafana]
+base_url="$GRAFANA_URL"
+dashboard_url="$GRAFANA_DASHBOARD_URL"
+EOF
+
+BASE64_CONFIG_OVERRIDE=$(cat config.toml | base64 -w 0)
+echo ::add-mask::$BASE64_CONFIG_OVERRIDE
+echo "BASE64_CONFIG_OVERRIDE=$BASE64_CONFIG_OVERRIDE" >> $GITHUB_ENV
+```
+
+**It is highly recommended to use reusable GHA actions present in [.actions](../../../.github/.actions) to generate and apply the base64-encoded configuration.** Own implementation of `BASE64_CONFIG_OVERRIDE` generation is discouraged and should be used only if existing actions do not cover the use case. But even in that case it might be a better idea to extend existing actions.
+
+This variable is automatically relayed to Kubernetes-based tests, eliminating the need for manual intervention in test scripts.
+
+The `overrides.toml` file is recommended for local use to adjust dynamic variables or modify predefined settings. At the very minimum it should contain the Chainlink image and version, as shown in the example below:
+
+```toml
+[ChainlinkImage]
+image = "your image name"
+version = "your tag"
+```
+
+Product-specific configurations, such as those in `[product_name].toml`, house the bulk of default and variant settings, supporting default configurations like the following in `log_poller.toml`:
+
+```toml
+# product defaults
+[LogPoller]
+[LogPoller.General]
+generator = "looped"
+contracts = 2
+events_per_tx = 4
+use_finality_tag = true
+log_poll_interval = "500ms"
+# 0 disables backup poller
+backup_log_poller_block_delay = 0
+
+[LogPoller.Looped]
+execution_count = 100
+min_emit_wait_time_ms = 200
+max_emit_wait_time_ms = 500
+```
+
+Named configurations allow for the customization of settings through unique identifiers, such as a test name or type, acting as specific overrides. Here's how you can define and use these configurations:
+
+For instance, to tailor configurations for a particular test, you might define it as follows:
+
+```toml
+# Here the configuration name is "TestLogManyFiltersPollerFinalityTag"
+[TestLogManyFiltersPollerFinalityTag.LogPoller.General]
+contracts = 300
+```
+
+Alternatively, for a configuration that applies to a certain type of test, as seen in `vrfv2.toml`, you could specify:
+
+```toml
+# Here the configuration name is "Soak"
+[Soak.VRFv2.Common]
+cancel_subs_after_test_run = true
+```
+
+When processing TOML files, the system initially searches for a general (unnamed) configuration. If a named configuration is found, it can specifically override the general (unnamed) settings, providing a targeted approach to configuration management based on distinct identifiers like test names or types.
+
+Finally `default.toml` file is envisioned to contain fundamental and universally applicable settings, such as logging configurations.
+
+## Local/Kubernetes Usage
+
+GitHub workflows in this repository have been updated to dynamically generate and utilize base64-encoded TOML configurations derived from user inputs or environment variables. For local execution or remote Kubernetes runners, users must manually supply certain variables, which cannot be embedded in configuration files due to their sensitive or dynamic nature.
+
+Essential variables might include:
+* Chainlink image and version
+* Test duration for specific tests (e.g., load, soak)
+* Configuration specific to Loki (mandatory for certain tests)
+* Grafana dashboard URLs
+
+For local testing, it is advisable to place these variables in the `overrides.toml` file. For Kubernetes or remote runners, the process involves creating a TOML file with the necessary values, encoding it in base64, and setting the result as the `BASE64_CONFIG_OVERRIDE` environment variable.
+
+## Embeded config
+Because Go automatically excludes TOML files during the compilation of binaries, we must take deliberate steps to include our configuration files in the compiled binary. This can be accomplished by using a custom build tag `-o embed`. Implementing this tag will incorporate all the default configurations located in the `./testconfig` folder directly into the binary. Therefore, when executing tests from the binary, you'll only need to supply the `overrides.toml` file. This file should list only the settings you wish to modify; all other configurations will be sourced from the embedded configurations. You can access these embedded configurations [here](.integration-tests/testconfig/configs_embed.go).
+
+## To bear in mind
+### Validation failures
+When the system encounters even a single setting related to a specific product or configuration within the configurations, it triggers a comprehensive validation of the entire configuration for that product. This approach is based on the assumption that if any configuration for a given product is specified, the entire set of configurations for that product must be complete and valid. This is particularly crucial when dealing with the `overrides.toml` file, where it's easy to overlook the need to comment out or adjust values when switching between configurations for different products. Essentially, the presence of any specific configuration detail necessitates that all relevant configurations for that product be fully defined and correct to prevent validation errors.
+
+## Possible nil pointers
+If no configuration values are set for a product or its logging parameters, the system won't perform validation checks. This can lead to a 'nil pointer exception' error if you attempt to access a configuration property later on. This situation arises because we use pointers to facilitate optional overrides; accessing an unset (nil) pointer will cause an error. To avoid such issues, especially when general validations might not cover every scenario, it's crucial for users to ensure that all necessary configuration options are explicitly set. Additionally, it's highly recommended to implement test-specific validations to confirm that all required values for a particular test are indeed established. This proactive approach helps prevent runtime errors and ensures smooth test execution.
+
+## Contributing
+It's crucial to incorporate all new test configuration settings directly into the TOML configuration files, steering clear of using environment variables for this purpose. Our goal is to centralize all configuration details, including examples, within the same package. This approach simplifies the process of understanding the available configuration options and identifying the appropriate values to use for each setting.
+
+## Reusing TestConfig in other projects
+To ensure the cleanliness and simplicity of your project's configuration, it's advised against using the `testconfig` code as a direct library in other projects. The reason is that much of this code is tailored specifically to its current application, which might not align with the requirements of your project. Your project might not necessitate any overrides or could perhaps benefit from a simpler configuration approach.
+
+However, if you find a need to utilize some methods from this project, the recommended practice is to implement the required interfaces within your project's configuration package, rather than directly copying and pasting code. For instance, if you aim to incorporate a setup action similar to the `SetupVRFV2Environment` for VRFv2, like the one shown below:
+
+```go
+func SetupVRFV2Environment(
+    env *test_env.CLClusterTestEnv,
+    nodesToCreate []vrfcommon.VRFNodeType,
+    vrfv2TestConfig types.VRFv2TestConfig,
+    useVRFOwner bool,
+    useTestCoordinator bool,
+    linkToken contracts.LinkToken,
+    mockNativeLINKFeed contracts.MockETHLINKFeed,
+    registerProvingKeyAgainstAddress string,
+    numberOfTxKeysToCreate int,
+    numberOfConsumers int,
+    numberOfSubToCreate int,
+    l zerolog.Logger,
+) (*vrfcommon.VRFContracts, []uint64, *vrfcommon.VRFKeyData, map[vrfcommon.VRFNodeType]*vrfcommon.VRFNode, error) {
+```
+
+You should not replicate the entire `TestConfig` structure. Instead, create an implementation of the `types.VRFv2TestConfig` interface in your project and use that as the parameter. This approach allows you to maintain a streamlined and focused configuration package in your project.
+
+## Known Issues/Limitations
+* Duplicate file names in different locations may lead to unpredictable configurations being selected.
+* The use of pointer fields for optional configuration elements necessitates careful handling, especially for programmatic modifications, to avoid unintended consequences. The `MustCopy()` function is recommended for creating deep copies of configurations for isolated modifications. Unfortunately some of the custom types are not copied at all, you need to set them manually. It's true for example for `blockchain.StrDuration` type.

integration-tests/testconfig/testconfig.go

@@ -384,6 +384,8 @@ func GetConfig(configurationName string, product Product) (TestConfig, error) {
 	logger.Debug().Msg("Validating test config")
 	err = testConfig.Validate()
 	if err != nil {
+		logger.Error().
+			Msg("Error validating test config. You might want refer to integration-tests/testconfig/README.md for more information.")
 		return TestConfig{}, errors.Wrapf(err, "error validating test config")
 	}
 
@@ -397,7 +399,7 @@ func GetConfig(configurationName string, product Product) (TestConfig, error) {
 
 func (c *TestConfig) readNetworkConfiguration() error {
 	// currently we need to read that kind of secrets only for network configuration
-	if c == nil {
+	if c.Network == nil {
 		c.Network = &ctf_config.NetworkConfig{}
 	}
 
@@ -421,19 +423,22 @@ func (c *TestConfig) Validate() error {
 			panic(fmt.Errorf("panic during test config validation: '%v'. Most probably due to presence of partial product config", r))
 		}
 	}()
+
 	if c.ChainlinkImage == nil {
-		return fmt.Errorf("chainlink image config must be set")
+		return MissingImageInfoAsError("chainlink image config must be set")
 	}
-	if err := c.ChainlinkImage.Validate(); err != nil {
-		return errors.Wrapf(err, "chainlink image config validation failed")
+	if c.ChainlinkImage != nil {
+		if err := c.ChainlinkImage.Validate(); err != nil {
+			return MissingImageInfoAsError(fmt.Sprintf("chainlink image config validation failed: %s", err.Error()))
+		}
 	}
 	if c.ChainlinkUpgradeImage != nil {
 		if err := c.ChainlinkUpgradeImage.Validate(); err != nil {
-			return errors.Wrapf(err, "chainlink upgrade image config validation failed")
+			return MissingImageInfoAsError(fmt.Sprintf("chainlink upgrade image config validation failed: %s", err.Error()))
 		}
 	}
 	if err := c.Network.Validate(); err != nil {
-		return errors.Wrapf(err, "network config validation failed")
+		return NoSelectedNetworkInfoAsError(fmt.Sprintf("network config validation failed: %s", err.Error()))
 	}
 
 	if c.Logging == nil {

integration-tests/testconfig/testconfig_utils.go

@@ -0,0 +1,70 @@
+package testconfig
+
+import (
+	"fmt"
+	"os"
+	"strings"
+)
+
+// MissingImageInfoAsError return a helfpul error message when the no Chainlink image info is found in TOML config.
+// If legacy env vars are found it prints ready to use TOML configuration
+func MissingImageInfoAsError(errStr string) error {
+	intro := `
+Old configuration approach detected. Please use TOML instead of env vars.
+Please refer to integration-tests/testconfig/README.md for more information.
+`
+
+	var imgStr, versionStr string
+
+	if img := os.Getenv("CHAINLINK_IMAGE"); img != "" {
+		imgStr = fmt.Sprintf("image = \"%s\"\n", img)
+	}
+
+	if version := os.Getenv("CHAINLINK_VERSION"); version != "" {
+		versionStr = fmt.Sprintf("version = \"%s\"\n", version)
+	}
+
+	finalErrStr := fmt.Sprintf("%s\n%s", errStr, intro)
+
+	if imgStr != "" && versionStr != "" {
+		extraInfo := `
+Or if you want to run your tests right now add following content to integration-tests/testconfig/overrides.toml:
+[ChainlinkImage]
+`
+		finalErrStr = fmt.Sprintf("%s\n%s%s%s%s", errStr, intro, extraInfo, imgStr, versionStr)
+	}
+
+	return fmt.Errorf(finalErrStr)
+}
+
+// NoSelectedNetworkInfoAsError return a helfpul error message when the no selected network info is found in TOML config.
+// If legacy env var is found it prints ready to use TOML configuration.
+func NoSelectedNetworkInfoAsError(errStr string) error {
+	intro := `
+Old configuration approach detected. Please use TOML instead of env vars.
+Please refer to integration-tests/testconfig/README.md for more information.
+`
+
+	finalErrStr := fmt.Sprintf("%s\n%s", errStr, intro)
+
+	if net := os.Getenv("SELECTED_NETWORKS"); net != "" {
+		parts := strings.Split(net, ",")
+		selectedNetworkStr := "["
+		for i, network := range parts {
+			selectedNetworkStr += fmt.Sprintf("\"%s\"", network)
+
+			if i < len(parts)-1 {
+				selectedNetworkStr += ", "
+			}
+		}
+		selectedNetworkStr += "]"
+
+		extraInfo := `
+Or if you want to run your tests right now add following content to integration-tests/testconfig/overrides.toml:
+[Network]
+selected_networks=`
+		finalErrStr = fmt.Sprintf("%s\n%s%s%s", errStr, intro, extraInfo, selectedNetworkStr)
+	}
+
+	return fmt.Errorf(finalErrStr)
+}