From 86e5a381b6a4ca25c3ba69351d08442b2ec640a6 Mon Sep 17 00:00:00 2001 From: Piotr Halama Date: Mon, 14 Oct 2024 07:31:28 +0200 Subject: [PATCH] Add warden documentation (#287) * Add warden documentation --------- Co-authored-by: Marcin Dobrochowski Co-authored-by: Grzegorz Karaluch --- .gitignore | 3 + docs/assets/user_architecture.svg | 4 ++ docs/assets/user_operations.svg | 4 ++ docs/assets/user_reconcile.svg | 4 ++ docs/contributor/00-01-overview-flow.md | 20 +++++++ docs/contributor/00-02-architecture.md | 35 +++++++++++ docs/contributor/01-10-configure_system.md | 31 ++++++++++ docs/user/00-01-overview-flow.md | 32 ++++++++++ docs/user/01-10-configure-user.md | 29 +++++++++ docs/user/README.md | 27 +++++++++ docs/user/_sidebar.md | 8 +++ .../01-20-use-warden-on-namespace.md | 59 +++++++++++++++++++ docs/user/tutorials/README.md | 3 + 13 files changed, 259 insertions(+) create mode 100644 docs/assets/user_architecture.svg create mode 100644 docs/assets/user_operations.svg create mode 100644 docs/assets/user_reconcile.svg create mode 100644 docs/contributor/00-01-overview-flow.md create mode 100644 docs/contributor/00-02-architecture.md create mode 100644 docs/contributor/01-10-configure_system.md create mode 100644 docs/user/00-01-overview-flow.md create mode 100644 docs/user/01-10-configure-user.md create mode 100644 docs/user/README.md create mode 100644 docs/user/_sidebar.md create mode 100644 docs/user/tutorials/01-20-use-warden-on-namespace.md create mode 100644 docs/user/tutorials/README.md diff --git a/.gitignore b/.gitignore index e5d9190c..75d57208 100644 --- a/.gitignore +++ b/.gitignore @@ -28,3 +28,6 @@ vendor # modularization generated files warden-manifest.yaml module-config.yaml + +# draw.io backup files +*.svg.bkp diff --git a/docs/assets/user_architecture.svg b/docs/assets/user_architecture.svg new file mode 100644 index 00000000..a0d0445c --- /dev/null +++ b/docs/assets/user_architecture.svg @@ -0,0 +1,4 @@ + + + +
Kyma Cluster
Kyma Cluster
Kyma User
Kyma User
Protected namespace
Protected namespace
Pod
Pod
2
2
3
3
1
1
applies
applies
Pod defintion
Pod defintion
Image
Image
Image
Image
Warden
Warden
Notary
Notary
Image signatures
Image signa...
Docker Registry
Docker Registry
Image
Image
Image
Image
triggers
triggers
retrieves
retrieves
4
4
retrieves
retrieves
creates
creates
5
5Text is not SVG - cannot display \ No newline at end of file diff --git a/docs/assets/user_operations.svg b/docs/assets/user_operations.svg new file mode 100644 index 00000000..0f52b94c --- /dev/null +++ b/docs/assets/user_operations.svg @@ -0,0 +1,4 @@ + + + +
Create Pod
Create Pod
Is the namespace protected?
Is the namespace p...
Is the image prefix allowed?
Is the image prefi...
No
No
Yes
Yes
No
No
Yes
Yes
No
No
No
No
No
No
Yes
Yes
Yes
Yes
Yes
Yes
Is the Notary server working?
Is the Notary serv...
Is the image signed?
Is the image signe...
Create Pod without validation
Create Pod without...
Is StrictMode enabled?
Is StrictMode enab...
Create Pod without validation, with the "pending" label
Create Pod without...
Reject Pod creation
Reject Pod creation
Create Pod with the "success" label
Create Pod with th...Text is not SVG - cannot display \ No newline at end of file diff --git a/docs/assets/user_reconcile.svg b/docs/assets/user_reconcile.svg new file mode 100644 index 00000000..f2ef8d8a --- /dev/null +++ b/docs/assets/user_reconcile.svg @@ -0,0 +1,4 @@ + + + +
Reconcile Pod
Reconcile Pod
Is the namespace protected?
Is the namespace p...
Is the image prefix allowed?
Is the image prefi...
No
No
Yes
Yes
No
No
Yes
Yes
Yes
Yes
Yes
Yes
No
No
No
No
Is the Notary server working?
Is the Notary serv...
Is the image signed?
Is the image signe...
Do nothing
Do nothing
Set the "pending" label on Pod
Set the "pending"...
Set the "failed" label on Pod
Set the "failed" l...
Set the "success" label on Pod
Set the "success"...Text is not SVG - cannot display \ No newline at end of file diff --git a/docs/contributor/00-01-overview-flow.md b/docs/contributor/00-01-overview-flow.md new file mode 100644 index 00000000..1e3de728 --- /dev/null +++ b/docs/contributor/00-01-overview-flow.md @@ -0,0 +1,20 @@ +# Warden Flow + +Warden works during the Pod create and update operations. It also reconciles Pods that are already running in the cluster. + +## Verification Modes + +Warden protects namespaces in two modes: system and user. +The system mode is designed for the internal Kyma operation namespaces. These namespaces are now marked (labeled) automatically by Lifecycle Manager (`kyma-system` namespace) and Istio (namespace for Istio). +The user mode can be enabled on any namespace and is controlled by the user. +Both of these modes are exclusive. + +Warden watches only namespaces with the `namespaces.warden.kyma-project.io/validate` label. +When such a label is changed, Warden reconfigures itself to work in the mode defined by the label. +After the label is removed, Warden stops watching the namespace and Pods in these namespaces are not verified anymore. + +For more information on configuring modes, see the [configuration](01-10-configure_system.md) section. + +## How It Works + +For the module overview, see [Warden Module](../user/README.md). diff --git a/docs/contributor/00-02-architecture.md b/docs/contributor/00-02-architecture.md new file mode 100644 index 00000000..f23e26b0 --- /dev/null +++ b/docs/contributor/00-02-architecture.md @@ -0,0 +1,35 @@ +# Warden Internal Architecture + +## Components + +Warden contains controllers and webhooks that are responsible for verification. + +### Pod Controller + +This controller checks Pods during the Pod update and periodic reconciliation. +The Pod controller uses an operation filter to exclude operations that are not relevant for verification. +For example, it does not verify Pods that changed unnecessary fields. Pods that changed images or validation status are verified. + +### Namespace Controller + +Namespace controller watches for namespace changes like changing the `namespaces.warden.kyma-project.io/validate` label annotations with user mode configuration. + +### Mutating Webhook + +Mutating webhook adds the `pods.warden.kyma-project.io/validate` label to the Pod. +It does the same operations as the Pod controller but additionally could decide to reject the Pod creation or update. For this purpose, it adds the internal `pods.warden.kyma-project.io/validate-reject: reject` annotation to the Pod. +This webhook also uses the strictMode configuration to decide if the Pod should be rejected when the Notary server is unavailable. + +Mutating webhook based on the current status of the Pod skips verification if the Pod is updating and its status is `pending` or `failed`. +It does this because the Pod controller previously set the status, and it is not necessary to verify the Pod again. + +### Validating Webhook + +Validation webhook only checks the `pods.warden.kyma-project.io/validate-reject: reject` annotation and rejects the Pod if it is present. + +## Image Verification + +Warden verifies that images used in Pods are signed by the Notary server by comparing the digest of the image in the Docker registry with the digest stored in the Notary server. +Warden checks if the checked artifact is an image or a list of images. If it is a list of images, Warden checks digest stored in Notary against the digest of the whole list. This is necessary, since calling the `remote.Image(ref)` method on a list of images returns only data for the first image in the list, which would allow tampering with the image list. + +If the artifact is an image, Warden checks the digest stored in Notary against the digest of the image. If that check fails, Warden makes a deprecated check against the image manifest digest. This check will be removed in the future. diff --git a/docs/contributor/01-10-configure_system.md b/docs/contributor/01-10-configure_system.md new file mode 100644 index 00000000..bb9a20d1 --- /dev/null +++ b/docs/contributor/01-10-configure_system.md @@ -0,0 +1,31 @@ +# System and User Configuration + +## System Configuration + +To enable Warden on a system namespace add the `namespaces.warden.kyma-project.io/validate: system` label to the namespace. +For now, we also support the deprecated `namespaces.warden.kyma-project.io/warden: enabled` equivalent label. + +The Warden configuration for the system mode is defined in [configmap.yaml](../../charts/warden/templates/configmap.yaml) in the section `data/config.yaml`. +You can set the following properties: + +| Name | Description | Default value | +|--------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|----------------------------------------------| +| `notary.URL` | URL of the Notary server used for image verification. | "https://signing-dev.repositories.cloud.sap" | +| `notary.allowedRegistries` | Comma-separated list of allowed registry prefixes. | "" | +| `notary.timeout` | Timeout for the Notary server connection. | "30s" | +| `admission.systemNamespace` | Namespace where the Warden admission controller is deployed. | "default" | +| `admission.serviceName` | Name of the Warden admission controller service. | "warden-admission" | +| `admission.secretName` | Name of the Secret containing the certificate for the Warden admission controller. | "warden-admission-cert" | +| `admission.port` | Port on which the Warden admission controller listens. | 8443 | +| `admission.timeout` | Timeout for the Warden admission controller. | "2s" | +| `admission.strictMode` | If set to `true`, Warden rejects all images when the Notary server is unavailable. If set to `false`, Warden adds the label `pods.warden.kyma-project.io/validate: pending` to the Pod and retries the validation later. | "false" | +| `operator.metricsBindAddress` | Address on which the Warden operator serves Prometheus metrics. | ":8080" | +| `operator.healthProbeBindAddress` | Address on which the Warden operator serves health probes. | ":8081" | +| `operator.leaderElect` | If set to `true`, Warden operator uses leader election for high availability. | false | +| `operator.podReconcilerRequeueAfter` | Time after which the pod reconciler re-queues the Pods that failed the validation. | "1h" | +| `logging.level` | Log level for Warden. | "info" | +| `logging.format` | Log format for Warden. | "text" | + +## User Configuration + +For the user configuration see [User Configuration](../user/01-10-configure-user.md). \ No newline at end of file diff --git a/docs/user/00-01-overview-flow.md b/docs/user/00-01-overview-flow.md new file mode 100644 index 00000000..be207af4 --- /dev/null +++ b/docs/user/00-01-overview-flow.md @@ -0,0 +1,32 @@ +# Warden Flow + +## Pod Create and Update Operations + +Warden validates images in Pods during the Pod create and update operations. It checks if the images are signed by the Notary server. If the images are not signed, Warden rejects the Pod creation or update. + +![Pod create and update flow](../assets/user_operations.svg) + +Strict mode determines if Warden must conditionally approve Pod when the Notary server is unavailable. If strict mode is enabled, Warden rejects all images when the Notary server is unavailable. If strict mode is disabled, Warden adds the `pods.warden.kyma-project.io/validate: pending` label to the Pod and retries validation later. + +## Pod Reconciliation + +Warden periodically reconciles Pods that are already running in the cluster. It checks if the images in the Pods are signed by the Notary server. If the images are not signed, Warden adds the `pods.warden.kyma-project.io/validate: failed` label to the Pod and retries validation later. + +![Pod reconciliation flow](../assets/user_reconcile.svg) + +Reconciliation can be triggered periodically or when the namespace is updated. + +## Namespace Update + +When a namespace has the `namespaces.warden.kyma-project.io/validate: user` label and the `namespaces.warden.kyma-project.io/notary-url` annotation, Warden protects all Pods in the namespace. + +When a protected namespace is updated, Warden schedules a reconciliation on all Pods in the namespace. See [Pod Reconciliation](#pod-reconciliation). + +For a complete list of configuration options, see [User Configuration](01-10-configure-user.md). + +## Verification Results + +Warden adds the labels `pods.warden.kyma-project.io/validate` to Pods to indicate the verification status: + * `success` - the Pod passed the controller check. + * `failed` - the Pod did not pass the controller check. + * `pending` - the verification status is unknown, and the Pod is waiting for validation. diff --git a/docs/user/01-10-configure-user.md b/docs/user/01-10-configure-user.md new file mode 100644 index 00000000..662edcb8 --- /dev/null +++ b/docs/user/01-10-configure-user.md @@ -0,0 +1,29 @@ +# User Configuration + +To enable the Warden module in your namespace, add the `namespaces.warden.kyma-project.io/validate: user` label to the namespace. +You can configure Warden on each namespace by adding the following annotations to the namespace: + +| Name | Required | Description | Default value | +| ------------------------------------------------------ | -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | +| `namespaces.warden.kyma-project.io/notary-url` | Yes | URL of the Notary server used for image verification. | "" | +| `namespaces.warden.kyma-project.io/allowed-registries` | No | Comma-separated list of allowed registry prefixes. | "" | +| `namespaces.warden.kyma-project.io/notary-timeout` | No | Timeout for the Notary server connection. | "30s" | +| `namespaces.warden.kyma-project.io/strict-mode` | No | If set to `true`, Warden rejects all images when the Notary server is unavailable. If set to `false`, Warden adds the label `pods.warden.kyma-project.io/validate: pending` to the Pod and retries the validation later. | "true" | + +# Example + +Example namespace configuration verified by Warden: + +```yaml +apiVersion: v1 +kind: Namespace +metadata: + name: my-namespace + labels: + namespaces.warden.kyma-project.io/validate: "user" + annotations: + namespaces.warden.kyma-project.io/notary-url: "https://notary.example.com" + namespaces.warden.kyma-project.io/allowed-registries: "registry1.io,registry2.io/nginx" + namespaces.warden.kyma-project.io/notary-timeout: "30s" + namespaces.warden.kyma-project.io/strict-mode: "true" +``` diff --git a/docs/user/README.md b/docs/user/README.md new file mode 100644 index 00000000..a94bc4bc --- /dev/null +++ b/docs/user/README.md @@ -0,0 +1,27 @@ +# Warden Module + +## Overview + +The Warden module verifies that images used in Pods are signed by the Notary server. +The module is enabled by default on BTP and configured to check the integrity of all Pods in the `kyma-system` namespace. +It offers two validation modes: user and system. The system validation is reserved for internal Kyma operation. The user validation can be enabled on any namespace and is controlled by the user. +For more information on the Warden module flow, see [Warden Flow](00-01-overview-flow.md) + +## Useful Links + +[User Configuration](tutorials/01-10-configure-user.md) + +## Warden Architecture + +![Architecture](../assets/user_architecture.svg) + +1. User applies Pod on a cluster to the protected namespace. +2. Kubernetes triggers Warden verification. +3. Warden retrieves the signature of the image or images used in the Pod from Notary. +4. Warden retrieves the image or images manifest from the Docker registry. +5. Warden allows (or disallows) creating Pod when the image is properly signed. + +### Image Verification + +Warden verifies that images used in Pods are signed by the Notary server by comparing the digest of the image in the Docker registry with the digest stored in the Notary server. +For multiplatform images, Warden verifies the digest of the index of images. diff --git a/docs/user/_sidebar.md b/docs/user/_sidebar.md new file mode 100644 index 00000000..054d5a14 --- /dev/null +++ b/docs/user/_sidebar.md @@ -0,0 +1,8 @@ + +* [Back to Kyma Home](/) +* [Warden Module](/warden/user/README.md) +* [User Configuration](/warden/user/01-10-configure-user.md) +* [Warden Flow](/warden/user/00-01-overview-flow.md) +* [Tutorials](/warden/user/tutorials/README.md) + * [Use Warden on a User Namespace](/warden/user/tutorials/01-20-use-warden-on-namespace.md) + diff --git a/docs/user/tutorials/01-20-use-warden-on-namespace.md b/docs/user/tutorials/01-20-use-warden-on-namespace.md new file mode 100644 index 00000000..33fdb6a2 --- /dev/null +++ b/docs/user/tutorials/01-20-use-warden-on-namespace.md @@ -0,0 +1,59 @@ +# Use Warden on a User Namespace + +This tutorial shows how to enable Warden validation on a namespace and validate the integrity of images in Pods. + +## Prerequisites + +Before you start, ensure that you have: + +- [kubectl](https://kubernetes.io/docs/tasks/tools/install-kubectl/). +- [Warden](../01-10-configure-user.md) enabled on your cluster. +- A namespace where you want to enable Warden validation. +- [Notary](https://github.com/notaryproject/notary) server instance. +- Signed images. + +## Steps + +1. Set up the following environment variables: + + ```bash + export NAMESPACE= + export SIGNED_POD_NAME= + export SIGNED_IMAGE= + export UNSIGNED_POD_NAME= + export UNSIGNED_IMAGE= + export NOTARY_URL= + ``` + +2. Enable the Warden validation on a namespace by adding the required `namespaces.warden.kyma-project.io/notary-url` annotation and the `namespaces.warden.kyma-project.io/validate: user` label to the namespace: + + ```bash + kubectl annotate namespace $NAMESPACE namespaces.warden.kyma-project.io/notary-url=$NOTARY_URL + kubectl label namespace $NAMESPACE namespaces.warden.kyma-project.io/validate=user + ``` + +> [!WARNING] +> If you add the label without the annotation, Warden will not validate images in the namespace. + +3. Create a Pod with the signed image: + + ```bash + kubectl run $SIGNED_POD_NAME --namespace $NAMESPACE --image $SIGNED_IMAGE + ``` + +4. Verify that the Pod has the `pods.warden.kyma-project.io/validate: success` label: + + ```bash + kubectl get pods $SIGNED_POD_NAME --namespace $NAMESPACE -o jsonpath='{.metadata.labels.pods\.warden\.kyma-project\.io/validate}' + ``` + + The output should be `success` if validation has succeeded. + +5. Try to create a Pod with the unsigned image: + + ```bash + kubectl run $UNSIGNED_POD_NAME --namespace $NAMESPACE --image $UNSIGNED_IMAGE + ``` + + You should get the following error: + `Error from server (Forbidden): admission webhook "validation.webhook.warden.kyma-project.io" denied the request: Pod images nginx:latest validation failed` diff --git a/docs/user/tutorials/README.md b/docs/user/tutorials/README.md new file mode 100644 index 00000000..5f1ef9d6 --- /dev/null +++ b/docs/user/tutorials/README.md @@ -0,0 +1,3 @@ +# Tutorials + +This section will help you understand how the Warden works and how to use it in different scenarios.