more docs

book/src/SUMMARY.md

@@ -2,14 +2,17 @@
 
 - [Overview](./overview.md)
 - [Framework](./framework/overview.md)
-    - [Overview](./framework/overview.md)
     - [Getting Started](./framework/getting_started.md)
     - [First Test](./framework/first_test.md)
     - [Connecting Chainlink Node](./framework/connecting_chainlink_node.md)
     - [Connecting Chainlink Node (Multiple networks)]()
-    - [Basic NodeSet Environment](./framework/basic_environment.md)
+    - [NodeSet Environment](./framework/nodeset_environment.md)
+    - [NodeSet with External Blockchain](./framework/nodeset_external.md)
+    - [NodeSet (Local Docker builds)](./framework/nodeset_docker_rebuild.md)
+    - [NodeSet Compat Environment](./framework/nodeset_compatibility.md)
     - [CLI](./framework/cli.md)
     - [Configuration](./framework/configuration.md)
+    - [Test Configuration](./framework/test_configuration_overrides.md)
     - [Secrets]()
     - [Observability Stack](framework/observability/observability_stack.md)
       - [Metrics]()
@@ -31,5 +34,6 @@
 
 ---
 
-- [Developing](./developing.md)
-- [Releasing modules](./releasing_modules.md)
+- [Developing](developing.md)
+  - [Components](developing/developing_components.md)
+- [Releasing modules](releasing_modules.md)

book/src/developing.md

@@ -1 +1,8 @@
 # Developing
+
+Here we describe good practices for developing components for our framework.
+
+Rules for components are simple:
+- Component should declare some `Input` and an optional `Output` (we use that so we can skip or cache any component results)
+- Components should be isolated, they should not return anything except basic types like `int`, `string`, `maps` or `structs`
+- Component **must** have documentation under [Components](./framework/components/overview.md), here is an [example](./framework/components/blockchains/anvil.md)

book/src/developing/developing_components.md

@@ -0,0 +1,96 @@
+## Developing Components
+
+To build a scalable framework that enables the reuse of our product deployments (contracts or services in Docker), we need to establish a clear component structure.
+
+```golang
+package mycomponent
+
+import (
+	"fmt"
+	"github.com/smartcontractkit/chainlink-testing-framework/framework"
+)
+
+type Input struct {
+    // inputs fields that component exposes for configuration
+    ...
+    // outputs are embedded into inputs so framework can automatically save them
+	Out                      *Output  `toml:"out"`
+}
+
+type Output struct {
+    UseCache bool             `toml:"use_cache"`
+    // outputs that will be dumped to config and cached
+}
+
+
+func NewComponent(input *Input) (*Output, error) {
+	if input.Out != nil && input.Out.UseCache {
+		return input.Out, nil
+	}
+
+	// component logic here
+	// deploy a docker container(s)
+	// or deploy a set of smart contracts
+
+	input.Out = &Output{
+	    UseCache: true,
+	    // other fields
+	    ...
+	}
+	return out, nil
+}
+```
+
+Each component can define inputs and outputs, following these rules:
+
+- Outputs should be included within inputs.
+- If your component is used for side effects output can be omitted.
+- `input.Out.UseCache` should be added if you'd like to use caching, see more [here](caching)
+
+### Docker components good practices for [testcontainers-go](https://golang.testcontainers.org/):
+
+An example [simple component](../../../../framework/components/blockchain/anvil.go)
+
+An example of [complex component](../../../../framework/components/clnode/clnode.go)
+
+An example of [composite component](../../../../framework/components/simple_node_set/node_set.go)
+
+- Inputs should include at least `image`, `tag` and `pull_image` field
+```golang
+	Image                string `toml:"image" validate:"required"`
+	Tag                  string `toml:"tag" validate:"required"`
+	PullImage            bool   `toml:"pull_image" validate:"required"`
+```
+
+- `ContainerRequest` must contain labels, network and alias required for local observability stack and deployment isolation
+```golang
+		Labels:   framework.DefaultTCLabels(),
+		Networks: []string{framework.DefaultNetworkName},
+		NetworkAliases: map[string][]string{
+			framework.DefaultNetworkName: {containerName},
+		},
+```
+- In order to copy files into container use `framework.WriteTmpFile(data string, fileName string)`
+```golang
+	userSecretsOverridesFile, err := WriteTmpFile(in.Node.UserSecretsOverrides, "user-secrets-overrides.toml")
+	if err != nil {
+		return nil, err
+	}
+```
+- Output of docker component must contain all the URLs component exposes for access, both for internal docker usage and external test (host) usage
+```golang
+	host, err := framework.GetHost(c)
+	if err != nil {
+		return nil, err
+	}
+	mp, err := c.MappedPort(ctx, nat.Port(bindPort))
+	if err != nil {
+		return nil, err
+	}
+
+	return &NodeOut{
+	    UseCache: true,
+		DockerURL: fmt.Sprintf("http://%s:%s", containerName, in.Node.Port),
+		HostURL:   fmt.Sprintf("http://%s:%s", host, mp.Port()),
+	}, nil
+```

book/src/framework/components/components.md → book/src/framework/components/developing_components.md

@@ -23,7 +23,7 @@ type Output struct {
 
 
 func NewComponent(input *Input) (*Output, error) {
-	if input.Out.UseCache {
+	if input.Out != nil && input.Out.UseCache {
 		return input.Out, nil
 	}
 	
@@ -47,11 +47,11 @@ Each component can define inputs and outputs, following these rules:
 
 ### Docker components good practices for [testcontainers-go](https://golang.testcontainers.org/):
 
-An example [simple component](components/blockchain/anvil.go)
+An example [simple component](../../../../framework/components/blockchain/anvil.go)
 
-An example of [complex component](components/clnode/clnode.go)
+An example of [complex component](../../../../framework/components/clnode/clnode.go)
 
-An example of [composite component](components/node_set_extended/don.go)
+An example of [composite component](../../../../framework/components/simple_node_set/node_set.go)
 
 - Inputs should include at least `image`, `tag` and `pull_image` field
 ```

book/src/framework/components/overview.md

@@ -1 +1,8 @@
 # Components
+
+CTF contains a lot of useful components, some of them are off-chain services like `Chainlink Node`, `NodeSet`
+
+CTF contains three groups of components:
+- Off-chain services like `CL Node`, `NodeSet`, `JobDistributor`, etc.
+- On-chain wrappers for [chainlink-deployments](https://github.com/smartcontractkit/chainlink-deployments) repository
+- Test components, blockchain simulators, fakes, etc

book/src/framework/connecting_chainlink_node.md

@@ -1,9 +1,6 @@
 # Connecting Chainlink Node
 
-The Chainlink Testing Framework (CTF) is a modular, data-driven tool that lets you explicitly define and configure various Chainlink components.
-
-Let's spin up a simple component.
-
+Now let's have an example of Chainlink node connected to some local blockchain.
 
 Create your configuration in `smoke.toml`
 ```toml
@@ -72,9 +69,15 @@ Select your configuration by setting `CTF_CONFIGS=smoke.toml` and run it
 go test -v -run TestNode
 ```
 
+Check `node url: ...` in logs, open it and login using default credentials:
+```
+[email protected]
+fj293fbBnlQ!f9vNs
+```
+
 Summary:
 - We defined configuration for `BlockchainNetwork` and `NodeWithDB` (Chainlink + PostgreSQL)
 - We connected them together by creating common network config in `NewNetworkCfgOneNetworkAllNodes`
-- We have a Chainlink node running, check `node url: ...` messages in logs to open UI
+- We explored the Chainlink node UI
 
-You can learn more about [component design](./components/overview.md) or proceed with another example of [connecting Chainlink node](./connecting_chainlink_node.md)
+Let's proceed with another example of [using node sets](./nodeset_environment)

book/src/framework/connecting_multiple_nodes.md

@@ -1,80 +1,14 @@
-# Writing your first test
-
-The Chainlink Testing Framework (CTF) is a modular, data-driven tool that lets you explicitly define and configure various Chainlink components.
-
-Let's spin up a simple component.
+# Connecting Multiple Blockchain Nodes
 
 
 Create your configuration in `smoke.toml`
 ```toml
-[blockchain_a]
-  chain_id = "31337"
-  image = "ghcr.io/gakonst/foundry:latest"
-  port = "8545"
-  type = "anvil"
-
-[cl_node]
-  data_provider_url = "http://example.com"
-
-  [cl_node.db]
-    image = "postgres:15.6"
-    pull_image = true
-
-  [cl_node.node]
-    image = "public.ecr.aws/chainlink/chainlink:v2.17.0"
-    pull_image = true
 ```
 
 Create your test in `smoke_test.go`
 ```golang
-
-package capabilities_test
-
-import (
-	"fmt"
-	"github.com/smartcontractkit/chainlink-testing-framework/framework"
-	"github.com/smartcontractkit/chainlink-testing-framework/framework/components/blockchain"
-	"github.com/smartcontractkit/chainlink-testing-framework/framework/components/clnode"
-	"github.com/stretchr/testify/require"
-	"testing"
-)
-
-type Config struct {
-	BlockchainA *blockchain.Input `toml:"blockchain_a" validate:"required"`
-	CLNode      *clnode.Input     `toml:"cl_node" validate:"required"`
-}
-
-func TestNode(t *testing.T) {
-	in, err := framework.Load[Config](t)
-	require.NoError(t, err)
-
-	bc, err := blockchain.NewBlockchainNetwork(in.BlockchainA)
-	require.NoError(t, err)
-
-	networkCfg, err := clnode.NewNetworkCfgOneNetworkAllNodes(bc)
-	require.NoError(t, err)
-	in.CLNode.Node.TestConfigOverrides = networkCfg
-
-	output, err := clnode.NewNodeWithDB(in.CLNode)
-	require.NoError(t, err)
-
-	t.Run("test something", func(t *testing.T) {
-		fmt.Printf("node url: %s\n", output.Node.HostURL)
-		require.NotEmpty(t, output.Node.HostURL)
-	})
-}
-
-
 ```
 
 Select your configuration by setting `CTF_CONFIGS=smoke.toml` and run it
 ```bash
-go test -v -run TestNode
-```
-
-Summary:
-- We defined configuration for `BlockchainNetwork` and `NodeWithDB` (Chainlink + PostgreSQL)
-- We connected them together by creating common network config in `NewNetworkCfgOneNetworkAllNodes`
-- We have a Chainlink node running, check `node url: ...` messages in logs to open UI
-
-You can learn more about [component design](./components/overview.md) or proceed with another example of [connecting Chainlink node](./connecting_chainlink_node.md)
+```