diff --git a/docs/docs/troubleshooting.md b/docs/docs/troubleshooting.md index 0818d70298..c899d3ef48 100644 --- a/docs/docs/troubleshooting.md +++ b/docs/docs/troubleshooting.md @@ -28,11 +28,8 @@ environment variables `CONTRAST_LOG_LEVEL`, `CONTRAST_LOG_FORMAT` and be enabled for logging, which are disabled by default. Subsystems include: `snp-issuer`, `kds-getter`, and `snp-validator`. To enable all subsystems, use `*` as the value for this environment variable. - -Warnings and error messages from subsystems get printed regardless of whether -the subsystem is listed in the `CONTRAST_LOG_SUBSYSTEMS` environment variable. - -#### Coordinator debug logging + Warnings and error messages from subsystems get printed regardless of whether + the subsystem is listed in the `CONTRAST_LOG_SUBSYSTEMS` environment variable. To configure debug logging with all subsystems for your Coordinator, add the following variables to your container definition. @@ -50,34 +47,123 @@ spec: # v1.PodSpec # ... ``` +:::info + +While the Contrast Coordinator has a policy that allows certain configurations, +the Initializer and service mesh don't. When changing environment variables of other +parts than the Coordinator, ensure to rerun `contrast generate` to update the policy. + +::: + To access the logs generated by the Coordinator, you can use `kubectl` with the -following command. +following command: ```sh kubectl logs ``` -#### Pod fails on startup +## Pod fails to start If the Coordinator or a workload pod fails to even start, it can be helpful to look at the events of the pod during the startup process using the `describe` command. ```sh -kubectl describe pod +kubectl -n events --for pod/ ``` Example output: ``` -Events: - Type Reason Age From Message - ---- ------ ---- ---- ------- - ... - Warning Failed 20s kubelet Error: failed to create containerd task: failed to create shim task: "CreateContainerRequest is blocked by policy: ... +LAST SEEN TYPE REASON OBJECT MESSAGE +32m Warning Failed Pod/coordinator-0 kubelet Error: failed to create containerd task: failed to create shim task: "CreateContainerRequest is blocked by policy: ... +``` + +A common error, as in this example, is that the container creation was blocked by the +policy. Potential reasons are a modification of the deployment YAML without updating +the policies afterwards, or a version mismatch between Contrast components. + +### Regenerating the policies + +To ensure there isn't a mismatch between Kubernetes resource YAML and the annotated +policies, rerun + +```sh +contrast generate ``` -In this example, the container creation was blocked by a policy. This suggests -that a policy hasn't been updated to accommodate recent changes to the -configuration. Make sure to run `contrast generate` when altering your -deployment. +on your deployment. If any of the policy annotations change, re-deploy with the updated policies. + +### Pin container images + +When generating the policies, Contrast will download the images specified in your deployment +YAML and include their cryptographic identity. If the image tag is moved to another +container image after the policy has been generated, the image downloaded at deploy time +will differ from the one at generation time, and the policy enforcement won't allow the +container to be started in the pod VM. + +To ensure the correct image is always used, pin the container image to a fixed `sha256`: + +```yaml +image: ubuntu:22.04@sha256:19478ce7fc2ffbce89df29fea5725a8d12e57de52eb9ea570890dc5852aac1ac +``` + +This way, the same image will still be pulled when the container tag (`22.04`) is moved +to another image. + +### Validate Contrast components match + +A version mismatch between Contrast components can cause policy validation or attestation +to fail. Each Contrast runtime is identifiable based on its (shortened) measurement value +used to name the runtime class version. + +First, analyze which runtime class is currently installed in your cluster by running + +```sh +kubectl get runtimeclasses +``` + +This should give you output similar to the following one. + +```sh +NAME HANDLER AGE +contrast-cc-30bfa8706b542271ec9b7762bbb400af contrast-cc-30bfa8706b542271ec9b7762bbb400af 23d +contrast-cc-4d70a6e266cca46dfa8e41d92874e638 contrast-cc-4d70a6e266cca46dfa8e41d92874e638 7d +contrast-cc-b817659e094106f61bf6c178c27153ba contrast-cc-b817659e094106f61bf6c178c27153ba 2d19h +contrast-cc-beee79ca916b9e5dc59602788cbfb097 contrast-cc-beee79ca916b9e5dc59602788cbfb097 121m +kata-cc-isolation kata-cc 45d +``` + +The output shows that there are four Contrast runtime classes installed (as well as the runtime class provided +by the AKS CoCo preview, which isn't used by Contrast). + +Next, check if the pod that won't start has the correct runtime class configured, and the +Coordinator uses the exact same runtime: + +```sh +kubectl -n get -o=jsonpath='{.spec.runtimeClassName}' pod/ +kubectl -n get -o=jsonpath='{.spec.runtimeClassName}' pod/ +``` + +The output should list the runtime class the pod is using: + +```sh +contrast-cc-beee79ca916b9e5dc59602788cbfb097 +contrast-cc-beee79ca916b9e5dc59602788cbfb097 +``` + +Version information about the currently used CLI can be obtained via the `version` flag: + +```sh +contrast --version +``` + +```sh +contrast version v0.X.0 + + runtime handler: contrast-cc-beee79ca916b9e5dc59602788cbfb097 + launch digest: beee79ca916b9e5dc59602788cbfb097721cde34943e1583a3918f21011a71c47f371f68e883f5e474a6d4053d931a35 + genpolicy version: 3.2.0.azl1.genpolicy0 + image versions: ghcr.io/edgelesssys/contrast/coordinator@sha256:... + ghcr.io/edgelesssys/contrast/initializer@sha256:... +```