docs: extend troubleshooting guide

docs/docs/troubleshooting.md

@@ -28,11 +28,8 @@ environment variables `CONTRAST_LOG_LEVEL`, `CONTRAST_LOG_FORMAT` and
   be enabled for logging, which are disabled by default. Subsystems include:
   `snp-issuer`, `kds-getter`, and `snp-validator`. To enable all subsystems, use
   `*` as the value for this environment variable.
-
-Warnings and error messages from subsystems get printed regardless of whether
-the subsystem is listed in the `CONTRAST_LOG_SUBSYSTEMS` environment variable.
-
-#### Coordinator debug logging
+  Warnings and error messages from subsystems get printed regardless of whether
+  the subsystem is listed in the `CONTRAST_LOG_SUBSYSTEMS` environment variable.
 
 To configure debug logging with all subsystems for your Coordinator, add the
 following variables to your container definition.
@@ -50,34 +47,123 @@ spec: # v1.PodSpec
     # ...
 ```
 
+:::info
+
+While the Contrast Coordinator has a policy that allows certain configurations,
+the Initializer and service mesh don't. When changing environment variables of other
+parts than the Coordinator, ensure to rerun `contrast generate` to update the policy.
+
+:::
+
 To access the logs generated by the Coordinator, you can use `kubectl` with the
-following command.
+following command:
 
 ```sh
 kubectl logs <coordinator-pod-name>
 ```
 
-#### Pod fails on startup
+## Pod fails to start
 
 If the Coordinator or a workload pod fails to even start, it can be helpful to
 look at the events of the pod during the startup process using the `describe`
 command.
 
 ```sh
-kubectl describe pod <pod-name>
+kubectl -n <namespace> events --for pod/<coordinator-pod-name>
 ```
 
 Example output:
 
 ```
-Events:
-  Type     Reason   Age    From     Message
-  ----     ------   ----   ----     -------
-  ...
-  Warning  Failed   20s    kubelet  Error: failed to create containerd task: failed to create shim task: "CreateContainerRequest is blocked by policy: ...
+LAST SEEN  TYPE     REASON  OBJECT             MESSAGE
+32m        Warning  Failed  Pod/coordinator-0  kubelet  Error: failed to create containerd task: failed to create shim task: "CreateContainerRequest is blocked by policy: ...
+```
+
+A common error, as in this example, is that the container creation was blocked by the
+policy. Potential reasons are a modification of the deployment YAML without updating
+the policies afterwards, or a version mismatch between Contrast components.
+
+### Regenerating the policies
+
+To ensure there isn't a mismatch between Kubernetes resource YAML and the annotated
+policies, rerun
+
+```sh
+contrast generate
 ```
 
-In this example, the container creation was blocked by a policy. This suggests
-that a policy hasn't been updated to accommodate recent changes to the
-configuration. Make sure to run `contrast generate` when altering your
-deployment.
+on your deployment. If any of the policy annotations change, re-deploy with the updated policies.
+
+### Pin container images
+
+When generating the policies, Contrast will download the images specified in your deployment
+YAML and include their cryptographic identity. If the image tag is moved to another
+container image after the policy has been generated, the image downloaded at deploy time
+will differ from the one at generation time, and the policy enforcement won't allow the
+container to be started in the pod VM.
+
+To ensure the correct image is always used, pin the container image to a fixed `sha256`:
+
+```yaml
+image: ubuntu:22.04@sha256:19478ce7fc2ffbce89df29fea5725a8d12e57de52eb9ea570890dc5852aac1ac
+```
+
+This way, the same image will still be pulled when the container tag (`22.04`) is moved
+to another image.
+
+### Validate Contrast components match
+
+A version mismatch between Contrast components can cause policy validation or attestation
+to fail. Each Contrast runtime is identifiable based on its (shortened) measurement value
+used to name the runtime class version.
+
+First, analyze which runtime class is currently installed in your cluster by running
+
+```sh
+kubectl get runtimeclasses
+```
+
+This should give you output similar to the following one.
+
+```sh
+NAME                                           HANDLER                                        AGE
+contrast-cc-30bfa8706b542271ec9b7762bbb400af   contrast-cc-30bfa8706b542271ec9b7762bbb400af   23d
+contrast-cc-4d70a6e266cca46dfa8e41d92874e638   contrast-cc-4d70a6e266cca46dfa8e41d92874e638   7d
+contrast-cc-b817659e094106f61bf6c178c27153ba   contrast-cc-b817659e094106f61bf6c178c27153ba   2d19h
+contrast-cc-beee79ca916b9e5dc59602788cbfb097   contrast-cc-beee79ca916b9e5dc59602788cbfb097   121m
+kata-cc-isolation                              kata-cc                                        45d
+```
+
+The output shows that there are four Contrast runtime classes installed (as well as the runtime class provided
+by the AKS CoCo preview, which isn't used by Contrast).
+
+Next, check if the pod that won't start has the correct runtime class configured, and the
+Coordinator uses the exact same runtime:
+
+```sh
+kubectl -n <namespace> get -o=jsonpath='{.spec.runtimeClassName}' pod/<pod-name>
+kubectl -n <namespace> get -o=jsonpath='{.spec.runtimeClassName}' pod/<coordinator-pod-name>
+```
+
+The output should list the runtime class the pod is using:
+
+```sh
+contrast-cc-beee79ca916b9e5dc59602788cbfb097
+contrast-cc-beee79ca916b9e5dc59602788cbfb097
+```
+
+Version information about the currently used CLI can be obtained via the `version` flag:
+
+```sh
+contrast --version
+```
+
+```sh
+contrast version v0.X.0
+
+    runtime handler:      contrast-cc-beee79ca916b9e5dc59602788cbfb097
+    launch digest:        beee79ca916b9e5dc59602788cbfb097721cde34943e1583a3918f21011a71c47f371f68e883f5e474a6d4053d931a35
+    genpolicy version:    3.2.0.azl1.genpolicy0
+    image versions:       ghcr.io/edgelesssys/contrast/coordinator@sha256:...
+                          ghcr.io/edgelesssys/contrast/initializer@sha256:...
+```