From 566fdc43e043e34f258e627583fb3b653d8bed29 Mon Sep 17 00:00:00 2001 From: Radek Scheibinger Date: Tue, 23 Apr 2024 15:44:26 +0200 Subject: [PATCH] Decouple crib config from cl-cluster helm chart (#12877) --- .gitignore | 1 + charts/chainlink-cluster/README.md | 156 +----------------- .../chainlink-cluster => crib}/.env.example | 0 crib/README.md | 156 ++++++++++++++++++ .../dashboard/cmd/delete.go | 0 .../dashboard/cmd/deploy.go | 0 .../chainlink-cluster => crib}/devspace.yaml | 4 +- .../devspace_start.sh | 0 {charts/chainlink-cluster => crib}/go.mod | 2 +- {charts/chainlink-cluster => crib}/go.sum | 0 .../scripts/ingress_check.sh | 0 {charts/chainlink-cluster => crib}/setup.sh | 0 dashboard-lib/README.md | 2 +- integration-tests/load/ocr/README.md | 2 +- 14 files changed, 165 insertions(+), 158 deletions(-) rename {charts/chainlink-cluster => crib}/.env.example (100%) create mode 100644 crib/README.md rename {charts/chainlink-cluster => crib}/dashboard/cmd/delete.go (100%) rename {charts/chainlink-cluster => crib}/dashboard/cmd/deploy.go (100%) rename {charts/chainlink-cluster => crib}/devspace.yaml (99%) rename {charts/chainlink-cluster => crib}/devspace_start.sh (100%) rename {charts/chainlink-cluster => crib}/go.mod (95%) rename {charts/chainlink-cluster => crib}/go.sum (100%) rename {charts/chainlink-cluster => crib}/scripts/ingress_check.sh (100%) rename {charts/chainlink-cluster => crib}/setup.sh (100%) diff --git a/.gitignore b/.gitignore index bcd526fbc43..0a938ff9123 100644 --- a/.gitignore +++ b/.gitignore @@ -25,6 +25,7 @@ tools/clroot/db.sqlite3-wal .env* .dbenv !charts/chainlink-cluster/.env.example +!crib/.env.example !.github/actions/setup-postgres/.env .direnv .idea diff --git a/charts/chainlink-cluster/README.md b/charts/chainlink-cluster/README.md index a0a4b8a78cc..b76d55090dd 100644 --- a/charts/chainlink-cluster/README.md +++ b/charts/chainlink-cluster/README.md @@ -2,112 +2,9 @@ Example CL nodes cluster for system level tests -Install `kubefwd` (no nixpkg for it yet, planned) - -``` -brew install txn2/tap/kubefwd -``` - -If you want to build images you need [docker](https://docs.docker.com/engine/install/) service running - -Enter the shell (from the root project dir) - -``` -nix develop -``` - # Develop - -## New cluster - -We are using [devspace](https://www.devspace.sh/docs/getting-started/installation?x0=3) - -Configure the cluster, see `deployments.app.helm.values` and [values.yaml](./values.yaml) comments for more details - -Set up your K8s access - -``` -export DEVSPACE_IMAGE="..." -./setup.sh ${my-personal-namespace-name-crib} -``` - -Create a .env file based on the .env.sample file - -```sh -cp .env.sample .env -# Fill in the required values in .env -``` - -Build and deploy the current state of your repository - -``` -devspace deploy -``` - -Default `ttl` is `72h`, use `ttl` command to update if you need more time - -Valid values are `1h`, `2m`, `3s`, etc. Go time format is invalid `1h2m3s` - -``` -devspace run ttl ${namespace} 120h -``` - -If you want to deploy an image tag that is already available in ECR, use: - -``` -devspace deploy --override-image-tag "" -``` - -If you want to deploy an image tag from a public ECR repo, use: - -``` -export DEVSPACE_IMAGE=public.ecr.aws/chainlink/chainlink -devspace deploy --override-image-tag 2.9.0 -``` - -Forward ports to check UI or run tests - -``` -devspace run connect ${my-personal-namespace-name-crib} -``` - -List ingress hostnames - -``` -devspace run ingress-hosts -``` - -Destroy the cluster - -``` -devspace purge -``` - -## Running load tests - -Check this [doc](../../integration-tests/load/ocr/README.md) - -If you used `devspace dev ...` always use `devspace reset pods` to switch the pods back - -# Helm - -If you would like to use `helm` directly, please uncomment data in `values.yaml` - -## Install from local files - -``` -helm install -f values.yaml cl-cluster . -``` - -Forward all apps (in another terminal) - -``` -sudo kubefwd svc -n cl-cluster -``` - -Then you can connect and run your tests - ## Install from release +Note: The setup below doesn't work at the moment. Add the repository @@ -130,6 +27,7 @@ helm install -f values.yaml cl-cluster . ``` ## Create a new release +Note: The setup below doesn't work at the moment. Bump version in `Chart.yml` add your changes and add `helm_release` label to any PR to trigger a release @@ -143,52 +41,4 @@ helm test cl-cluster ``` helm uninstall cl-cluster -``` - -# Grafana dashboard - -We are using [Grabana](https://github.com/K-Phoen/grabana) lib to create dashboards programmatically - -You can also select dashboard platform in `INFRA_PLATFORM` either `kubernetes` or `docker` - -You can select the dashboard panels with `PANELS_INCLUDED` which is a list of panel names separated by comma -If you don't specify it will include core panels by default - -``` -export LOKI_TENANT_ID=promtail -export LOKI_URL=... -export GRAFANA_URL=... -export GRAFANA_TOKEN=... -export PROMETHEUS_DATA_SOURCE_NAME=Thanos -export LOKI_DATA_SOURCE_NAME=Loki -export INFRA_PLATFORM=kubernetes -export GRAFANA_FOLDER=DashboardCoreDebug -export DASHBOARD_NAME=CL-Cluster - -devspace run dashboard_deploy -``` - -Open Grafana folder `DashboardCoreDebug` and find dashboard `ChainlinkClusterDebug` - -# Testing - -Deploy your dashboard and run soak/load [tests](../../integration-tests/load/), check [README](../../integration-tests/README.md) for further explanations - -``` -devspace run dashboard_deploy -devspace run workload -devspace run dashboard_test -``` - -# Local Testing - -Go to [dashboard-lib](../../dashboard) and link the modules locally - -``` -cd dashboard -pnpm link --global -cd charts/chainlink-cluster/dashboard/tests -pnpm link --global dashboard-tests -``` - -Then run the tests with commands mentioned above +``` \ No newline at end of file diff --git a/charts/chainlink-cluster/.env.example b/crib/.env.example similarity index 100% rename from charts/chainlink-cluster/.env.example rename to crib/.env.example diff --git a/crib/README.md b/crib/README.md new file mode 100644 index 00000000000..c9106d401cd --- /dev/null +++ b/crib/README.md @@ -0,0 +1,156 @@ +# Crib Devspace Setup + +CRIB is a devspace configuration to launch chainlink cluster for system level tests + +Install `kubefwd` (no nixpkg for it yet, planned) + +``` +brew install txn2/tap/kubefwd +``` + +If you want to build images you need [docker](https://docs.docker.com/engine/install/) service running + +Enter the shell (from the root project dir) + +``` +nix develop +``` + +# Develop + +## New cluster + +We are using [devspace](https://www.devspace.sh/docs/getting-started/installation?x0=3) + +Configure the cluster, see `deployments.app.helm.values` and [values.yaml](../charts/chainlink-cluster/values.yaml) comments for more details + +Set up your K8s access + +``` +export DEVSPACE_IMAGE="..." +./setup.sh ${my-personal-namespace-name-crib} +``` + +Create a .env file based on the .env.sample file + +```sh +cp .env.sample .env +# Fill in the required values in .env +``` + +Build and deploy the current state of your repository + +``` +devspace deploy +``` + +Default `ttl` is `72h`, use `ttl` command to update if you need more time + +Valid values are `1h`, `2m`, `3s`, etc. Go time format is invalid `1h2m3s` + +``` +devspace run ttl ${namespace} 120h +``` + +If you want to deploy an image tag that is already available in ECR, use: + +``` +devspace deploy --override-image-tag "" +``` + +If you want to deploy an image tag from a public ECR repo, use: + +``` +export DEVSPACE_IMAGE=public.ecr.aws/chainlink/chainlink +devspace deploy --override-image-tag 2.9.0 +``` + +Forward ports to check UI or run tests + +``` +devspace run connect ${my-personal-namespace-name-crib} +``` + +List ingress hostnames + +``` +devspace run ingress-hosts +``` + +Destroy the cluster + +``` +devspace purge +``` + +## Running load tests + +Check this [doc](../../integration-tests/load/ocr/README.md) + +If you used `devspace dev ...` always use `devspace reset pods` to switch the pods back + +# Helm + +If you would like to use `helm` directly, please uncomment data in `values.yaml` + +## Install from local files + +``` +helm install -f values.yaml cl-cluster . +``` + +Forward all apps (in another terminal) + +``` +sudo kubefwd svc -n cl-cluster +``` + +Then you can connect and run your tests + +# Grafana dashboard + +We are using [Grabana](https://github.com/K-Phoen/grabana) lib to create dashboards programmatically + +You can also select dashboard platform in `INFRA_PLATFORM` either `kubernetes` or `docker` + +You can select the dashboard panels with `PANELS_INCLUDED` which is a list of panel names separated by comma +If you don't specify it will include core panels by default + +``` +export LOKI_TENANT_ID=promtail +export LOKI_URL=... +export GRAFANA_URL=... +export GRAFANA_TOKEN=... +export PROMETHEUS_DATA_SOURCE_NAME=Thanos +export LOKI_DATA_SOURCE_NAME=Loki +export INFRA_PLATFORM=kubernetes +export GRAFANA_FOLDER=DashboardCoreDebug +export DASHBOARD_NAME=CL-Cluster + +devspace run dashboard_deploy +``` + +Open Grafana folder `DashboardCoreDebug` and find dashboard `ChainlinkClusterDebug` + +# Testing + +Deploy your dashboard and run soak/load [tests](../../integration-tests/load/), check [README](../../integration-tests/README.md) for further explanations + +``` +devspace run dashboard_deploy +devspace run workload +devspace run dashboard_test +``` + +# Local Testing + +Go to [dashboard-lib](../dashboard-lib) and link the modules locally + +``` +cd dashboard +pnpm link --global +cd crib/dashboard/tests +pnpm link --global dashboard-tests +``` + +Then run the tests with commands mentioned above diff --git a/charts/chainlink-cluster/dashboard/cmd/delete.go b/crib/dashboard/cmd/delete.go similarity index 100% rename from charts/chainlink-cluster/dashboard/cmd/delete.go rename to crib/dashboard/cmd/delete.go diff --git a/charts/chainlink-cluster/dashboard/cmd/deploy.go b/crib/dashboard/cmd/deploy.go similarity index 100% rename from charts/chainlink-cluster/dashboard/cmd/deploy.go rename to crib/dashboard/cmd/deploy.go diff --git a/charts/chainlink-cluster/devspace.yaml b/crib/devspace.yaml similarity index 99% rename from charts/chainlink-cluster/devspace.yaml rename to crib/devspace.yaml index d46e28572bb..ca12e1ed205 100644 --- a/charts/chainlink-cluster/devspace.yaml +++ b/crib/devspace.yaml @@ -52,7 +52,7 @@ commands: ttl: |- kubectl label namespace $1 cleanup.kyverno.io/ttl=$2 --overwrite workload: |- - cd ../../integration-tests/load/ocr && go test -v -run TestOCRLoad || cd - + cd ../integration-tests/load/ocr && go test -v -run TestOCRLoad || cd - dashboard_deploy: |- go run dashboard/cmd/deploy.go dashboard_test: |- @@ -98,7 +98,7 @@ deployments: releaseName: "app" chart: name: cl-cluster - path: . + path: ../charts/chainlink-cluster # for simplicity, we define all the values here # they can be defined the same way in values.yml # devspace merges these "values" with the "values.yaml" before deploy diff --git a/charts/chainlink-cluster/devspace_start.sh b/crib/devspace_start.sh similarity index 100% rename from charts/chainlink-cluster/devspace_start.sh rename to crib/devspace_start.sh diff --git a/charts/chainlink-cluster/go.mod b/crib/go.mod similarity index 95% rename from charts/chainlink-cluster/go.mod rename to crib/go.mod index 4a8dd43fd5f..28bcaece757 100644 --- a/charts/chainlink-cluster/go.mod +++ b/crib/go.mod @@ -34,5 +34,5 @@ replace ( github.com/mwitkow/grpc-proxy => github.com/smartcontractkit/grpc-proxy v0.0.0-20230731113816-f1be6620749f github.com/sercand/kuberesolver/v4 => github.com/sercand/kuberesolver/v5 v5.1.1 - github.com/smartcontractkit/chainlink/dashboard-lib => ../../dashboard-lib + github.com/smartcontractkit/chainlink/dashboard-lib => ./../dashboard-lib ) diff --git a/charts/chainlink-cluster/go.sum b/crib/go.sum similarity index 100% rename from charts/chainlink-cluster/go.sum rename to crib/go.sum diff --git a/charts/chainlink-cluster/scripts/ingress_check.sh b/crib/scripts/ingress_check.sh similarity index 100% rename from charts/chainlink-cluster/scripts/ingress_check.sh rename to crib/scripts/ingress_check.sh diff --git a/charts/chainlink-cluster/setup.sh b/crib/setup.sh similarity index 100% rename from charts/chainlink-cluster/setup.sh rename to crib/setup.sh diff --git a/dashboard-lib/README.md b/dashboard-lib/README.md index 44fd655c72a..ee8fa66a7c7 100644 --- a/dashboard-lib/README.md +++ b/dashboard-lib/README.md @@ -16,7 +16,7 @@ dashboard Each component should contain rows, logic and unique variables in `component.go` -Components should be imported from this module, see [example](../charts/chainlink-cluster/dashboard/cmd/deploy.go) +Components should be imported from this module, see [example](../crib/dashboard/cmd/deploy.go) ## How to convert from JSON using Grabana codegen utility 1. Download Grabana binary [here](https://github.com/K-Phoen/grabana/releases) diff --git a/integration-tests/load/ocr/README.md b/integration-tests/load/ocr/README.md index 61951ba700f..e63828a7d93 100644 --- a/integration-tests/load/ocr/README.md +++ b/integration-tests/load/ocr/README.md @@ -13,7 +13,7 @@ sudo kubefwd svc -n cl-cluster Change environment connection configuration [here](../../../charts/chainlink-cluster/connect.toml) -If you haven't changed anything in [devspace.yaml](../../../charts/chainlink-cluster/devspace.yaml) then default connection configuration will work +If you haven't changed anything in [devspace.yaml](../../crib/devspace.yaml) then default connection configuration will work ## Usage