-
Notifications
You must be signed in to change notification settings - Fork 41
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
- Loading branch information
Showing
35 changed files
with
84 additions
and
1,055 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,15 +1,18 @@ | ||
| ## MDBook | ||
| Install [mdbook](https://github.com/rust-lang/mdBook) | ||
| ``` | ||
| # Chainlink Testing Framework Docs | ||
|
|
||
| We use [mdBook](https://github.com/rust-lang/mdBook) for our docs. They can be found [here](https://smartcontractkit.github.io/chainlink-testing-framework/). | ||
|
|
||
| ## Development | ||
|
|
||
| First [install Rust](https://doc.rust-lang.org/cargo/getting-started/installation.html), then [mdbook](https://github.com/rust-lang/mdBook). | ||
|
|
||
| ```sh | ||
| # Install mdBook | ||
| cargo install mdbook && \ | ||
| cargo install mdbook-alerts && \ | ||
| cargo install mdbook-cmdrun | ||
| ``` | ||
|
|
||
| Run it locally | ||
| ``` | ||
| # Run the mdBook | ||
| make run | ||
| ``` | ||
| Open your browser [here](http://localhost:9999/) | ||
|
|
||
| GitHub hosted [version](https://smartcontractkit.github.io/chainlink-testing-framework/overview.html) | ||
| Visit [localhost:9999](http://localhost:9999) to see your local version. |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,211 +1,41 @@ | ||
| ## Havoc | ||
| # Havoc | ||
|
|
||
| The `havoc` package is a Go library designed to facilitate chaos testing within Kubernetes environments using Chaos Mesh. It offers a structured way to define, execute, and manage chaos experiments as code, directly integrated into Go applications or testing suites. This package simplifies the creation and control of Chaos Mesh experiments, including network chaos, pod failures, and stress testing on Kubernetes clusters. | ||
| [](https://pkg.go.dev/github.com/smartcontractkit/chainlink-testing-framework/havoc) | ||
|
|
||
| ### Features | ||
| The `havoc` package is designed to facilitate chaos testing within [Kubernetes](https://kubernetes.io/) environments using [Chaos Mesh](https://chaos-mesh.org/). It offers a structured way to define, execute, and manage chaos experiments as code, directly integrated into Go applications or testing suites, simplifying the creation and control of Chaos Mesh experiments. | ||
|
|
||
| - **Chaos Object Management:** Easily create, update, pause, resume, and delete chaos experiments using Go structures and methods. | ||
| - **Lifecycle Hooks:** Utilize chaos listeners to hook into lifecycle events of chaos experiments, such as creation, start, pause, resume, and finish. | ||
| - **Support for Various Chaos Experiments:** Create and manage different types of chaos experiments like NetworkChaos, IOChaos, StressChaos, PodChaos, and HTTPChaos. | ||
| - **Chaos Experiment Status Monitoring:** Monitor and react to the status of chaos experiments programmatically. | ||
| ## Features | ||
|
|
||
| ### Installation | ||
| - **Chaos Object Management:** Create, update, pause, resume, and delete chaos experiments using Go structures and methods. | ||
| - **Lifecycle Hooks:** Utilize chaos listeners to hook into the lifecycle of chaos experiments. | ||
| - **Different Experiments:** Create and manage different types of chaos experiments to affect network, IO, K8s pods, and more. | ||
| - **Active Monitoring:** Monitor and react to the status of chaos experiments programmatically. | ||
|
|
||
| To use `havoc` in your project, ensure you have a Go environment setup. Then, install the package using go get: | ||
| ## Requirements | ||
|
|
||
| ``` | ||
| go get -u github.com/smartcontractkit/chainlink-testing-framework/havoc | ||
| ``` | ||
|
|
||
| Ensure your Kubernetes cluster is accessible and that you have Chaos Mesh installed and configured. | ||
|
|
||
| ### Monitoring and Observability in Chaos Experiments | ||
|
|
||
| `havoc` enhances chaos experiment observability through structured logging and Grafana annotations, facilitated by implementing the ChaosListener interface. This approach allows for detailed monitoring, debugging, and visual representation of chaos experiments' impact. | ||
|
|
||
| #### Structured Logging with ChaosLogger | ||
| - [Go](https://go.dev/) | ||
| - A Kubernetes cluster with [Chaos Mesh installed](https://chaos-mesh.org/docs/quick-start/) | ||
|
|
||
| `ChaosLogger` leverages the zerolog library to provide structured, queryable logging of chaos events. It automatically logs key lifecycle events such as creation, start, pause, and termination of chaos experiments, including detailed contextual information. | ||
| ## Installation | ||
|
|
||
| Instantiate `ChaosLogger` and register it as a listener to your chaos experiments: | ||
|
|
||
| ``` | ||
| logger := havoc.NewChaosLogger() | ||
| chaos.AddListener(logger) | ||
| ```sh | ||
| go get -u github.com/smartcontractkit/chainlink-testing-framework/havoc | ||
| ``` | ||
|
|
||
| ### Default package logger | ||
| ## Active Monitoring | ||
|
|
||
| `havoc/logger.go` contains default `Logger` instance for the package. | ||
| `havoc` enhances chaos experiment observability through structured logging and Grafana annotations by implementing the [ChaosListener](https://pkg.go.dev/github.com/smartcontractkit/chainlink-testing-framework/havoc#ChaosListener) interface. | ||
|
|
||
| #### Visual Monitoring with Grafana Annotations | ||
| The [ChaosLogger](https://pkg.go.dev/github.com/smartcontractkit/chainlink-testing-framework/havoc#ChaosLogger) is the default implementation. It uses [zerolog](https://github.com/rs/zerolog) to provide structured, queryable logging of chaos events. It automatically logs key lifecycle events such as creation, start, pause, and termination of chaos experiments with detailed contextual information. | ||
|
|
||
| `SingleLineGrafanaAnnotator` is a `ChaosListener` that annotates Grafana dashboards with chaos experiment events. This visual representation helps correlate chaos events with their effects on system metrics and logs. | ||
| ### Grafana Annotations | ||
|
|
||
| Initialize `SingleLineGrafanaAnnotator` with your Grafana instance details and register it alongside `ChaosLogger`: | ||
| We recommend using [Grafana dashboards](https://grafana.com/) to monitor your chaos experiments, and provide the [SingleLineGrafanaAnnotator](https://pkg.go.dev/github.com/smartcontractkit/chainlink-testing-framework/havoc#SingleLineGrafanaAnnotator), a `ChaosListener` that annotates dashboards with chaos experiment events so you can see in real time what your chaos experiment is doing. | ||
|
|
||
| ```golang | ||
| annotator := havoc.NewSingleLineGrafanaAnnotator( | ||
| "http://grafana-instance.com", | ||
| "grafana-access-token", | ||
| "dashboard-uid", | ||
| ) | ||
| chaos.AddListener(annotator) | ||
| ``` | ||
| You can also use the [RangeGrafanaAnnotator](https://pkg.go.dev/github.com/smartcontractkit/chainlink-testing-framework/havoc#RangeGrafanaAnnotator) to show the full range of a chaos event's duration rather than a single line. | ||
|
|
||
| ### Creating a Chaos Experiment | ||
| ## Creating a Chaos Experiment | ||
|
|
||
| To create a chaos experiment, define the chaos object options, initialize a chaos experiment with NewChaos, and then call Create to start the experiment. | ||
|
|
||
| Here is an example of creating and starting a PodChaos experiment: | ||
|
|
||
| ```golang | ||
| package main | ||
|
|
||
| import ( | ||
| "context" | ||
| "github.com/smartcontractkit/chainlink-testing-framework/havoc" | ||
| "github.com/chaos-mesh/chaos-mesh/api/v1alpha1" | ||
| "sigs.k8s.io/controller-runtime/pkg/client" | ||
| "time" | ||
| ) | ||
|
|
||
| func main() { | ||
| // Initialize dependencies | ||
| client, err := havoc.NewChaosMeshClient() | ||
| if err != nil { | ||
| panic(err) | ||
| } | ||
| logger := havoc.NewChaosLogger() | ||
| annotator := havoc.NewSingleLineGrafanaAnnotator( | ||
| "http://grafana-instance.com", | ||
| "grafana-access-token", | ||
| "dashboard-uid", | ||
| ) | ||
|
|
||
| // Define chaos experiment | ||
| podChaos := &v1alpha1.PodChaos{ /* PodChaos spec */ } | ||
| chaos, err := havoc.NewChaos(havoc.ChaosOpts{ | ||
| Object: podChaos, | ||
| Description: "Pod failure example", | ||
| DelayCreate: 5 * time.Second, | ||
| Client: client, | ||
| }) | ||
| if err != nil { | ||
| panic(err) | ||
| } | ||
|
|
||
| // Register listeners | ||
| chaos.AddListener(logger) | ||
| chaos.AddListener(annotator) | ||
|
|
||
| // Start chaos experiment | ||
| chaos.Create(context.Background()) | ||
|
|
||
| // Manage chaos lifecycle... | ||
| } | ||
| ``` | ||
|
|
||
| ### Test Example | ||
|
|
||
| ```golang | ||
| func TestChaosDON(t *testing.T) { | ||
| testDuration := time.Minute * 60 | ||
|
|
||
| // Load test config | ||
| cfg := &config.MercuryQAEnvChaos{} | ||
|
|
||
| // Define chaos experiments and their schedule | ||
|
|
||
| k8sClient, err := havoc.NewChaosMeshClient() | ||
| require.NoError(t, err) | ||
|
|
||
| // Test 3.2: Disable 2 nodes simultaneously | ||
|
|
||
| podFailureChaos4, err := k8s_chaos.MercuryPodChaosSchedule(k8s_chaos.MercuryScheduledPodChaosOpts{ | ||
| Name: "schedule-don-ocr-node-failure-4", | ||
| Description: "Disable 2 nodes (clc-ocr-mercury-arb-testnet-qa-nodes-3 and clc-ocr-mercury-arb-testnet-qa-nodes-4)", | ||
| DelayCreate: time.Minute * 0, | ||
| Duration: time.Minute * 20, | ||
| Namespace: cfg.ChaosNodeNamespace, | ||
| PodSelector: v1alpha1.PodSelector{ | ||
| Mode: v1alpha1.AllMode, | ||
| Selector: v1alpha1.PodSelectorSpec{ | ||
| GenericSelectorSpec: v1alpha1.GenericSelectorSpec{ | ||
| Namespaces: []string{cfg.ChaosNodeNamespace}, | ||
| ExpressionSelectors: v1alpha1.LabelSelectorRequirements{ | ||
| { | ||
| Key: "app.kubernetes.io/instance", | ||
| Operator: "In", | ||
| Values: []string{ | ||
| "clc-ocr-mercury-arb-testnet-qa-nodes-3", | ||
| "clc-ocr-mercury-arb-testnet-qa-nodes-4", | ||
| }, | ||
| }, | ||
| }, | ||
| }, | ||
| }, | ||
| }, | ||
| Client: k8sClient, | ||
| }) | ||
| require.NoError(t, err) | ||
|
|
||
| // Test 3.3: Disable 3 nodes simultaneously | ||
|
|
||
| podFailureChaos5, err := k8s_chaos.MercuryPodChaosSchedule(k8s_chaos.MercuryScheduledPodChaosOpts{ | ||
| Name: "schedule-don-ocr-node-failure-5", | ||
| Description: "Disable 3 nodes (clc-ocr-mercury-arb-testnet-qa-nodes-3, clc-ocr-mercury-arb-testnet-qa-nodes-4 and clc-ocr-mercury-arb-testnet-qa-nodes-5)", | ||
| DelayCreate: time.Minute * 40, | ||
| Duration: time.Minute * 20, | ||
| Namespace: cfg.ChaosNodeNamespace, | ||
| PodSelector: v1alpha1.PodSelector{ | ||
| Mode: v1alpha1.AllMode, | ||
| Selector: v1alpha1.PodSelectorSpec{ | ||
| GenericSelectorSpec: v1alpha1.GenericSelectorSpec{ | ||
| Namespaces: []string{cfg.ChaosNodeNamespace}, | ||
| ExpressionSelectors: v1alpha1.LabelSelectorRequirements{ | ||
| { | ||
| Key: "app.kubernetes.io/instance", | ||
| Operator: "In", | ||
| Values: []string{ | ||
| "clc-ocr-mercury-arb-testnet-qa-nodes-3", | ||
| "clc-ocr-mercury-arb-testnet-qa-nodes-4", | ||
| "clc-ocr-mercury-arb-testnet-qa-nodes-5", | ||
| }, | ||
| }, | ||
| }, | ||
| }, | ||
| }, | ||
| }, | ||
| Client: k8sClient, | ||
| }) | ||
| require.NoError(t, err) | ||
|
|
||
| chaosList := []havoc.ChaosEntity{ | ||
| podFailureChaos4, | ||
| podFailureChaos5, | ||
| } | ||
|
|
||
| for _, chaos := range chaosList { | ||
| chaos.AddListener(havoc.NewChaosLogger()) | ||
| chaos.AddListener(havoc.NewSingleLineGrafanaAnnotator(cfg.GrafanaURL, cfg.GrafanaToken, cfg.GrafanaDashboardUID)) | ||
|
|
||
| // Fail the test if the chaos object already exists | ||
| exists, err := havoc.ChaosObjectExists(chaos.GetObject(), k8sClient) | ||
| require.NoError(t, err) | ||
| require.False(t, exists, "chaos object already exists: %s. Delete it before starting the test", chaos.GetChaosName()) | ||
|
|
||
| chaos.Create(context.Background()) | ||
| } | ||
|
|
||
| t.Cleanup(func() { | ||
| for _, chaos := range chaosList { | ||
| // Delete chaos object if it still exists | ||
| chaos.Delete(context.Background()) | ||
| } | ||
| }) | ||
|
|
||
| // Simulate user activity/load for the duration of the chaos experiments | ||
| runUserLoad(t, cfg, testDuration) | ||
| } | ||
| ``` | ||
| See [this runnable example](https://pkg.go.dev/github.com/smartcontractkit/chainlink-testing-framework/havoc#ExampleNewChaos) of defining a chaos experiment. |
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
This file was deleted.
Oops, something went wrong.
Oops, something went wrong.