From d11f27c01ec0635b2eaa9d70e56087ca8e48c3da Mon Sep 17 00:00:00 2001 From: Markus Rudy Date: Mon, 29 Jan 2024 18:30:11 +0100 Subject: [PATCH] fixup! describe user workflow in README --- README.md | 171 +++++++++++++++++++++++++++++++++++------------------- 1 file changed, 111 insertions(+), 60 deletions(-) diff --git a/README.md b/README.md index 4775f88aa..4fa133ccb 100644 --- a/README.md +++ b/README.md @@ -29,95 +29,146 @@ fetches the workload certificate. The Initializer runs as init container before ## Generic Workflow -### Setup +### Prerequisites -First of all, you will need a Nunki binary and a workspace directory. At this stage of the project, -you will need to follow the instructions in [CONTRIBUTING.md](CONTRIBUTING.md) to set up a -development environment and then run `just demodir` to initialize a workspace. In the future, this -step may become part of the released Nunki binary. +* An empty workspace directory. +* The `nunki` binary. + + +```sh +WORKSPACE=$(mktemp -d) +cd "$WORKSPACE" +curl -Lo nunki https://... +``` ### Kubernetes Resources -In this step, you will define Kubernetes resources for the Nunki coordinator and the workloads you -want to run. Nunki works with vanilla Kubernetes yaml, so all you have to do is copy the resource -files into a `resources` sub-directory of your workspace. If you use Helm, you can do that with -`helm template`; if you use Kustomize, you can do that with `kustomize build`. +All Kubernetes resources that should be running in confidential containers must be present in the +resources directory. You can generate them from a Helm chart or from a Kustomization, or just copy +them over from your repository. + +```sh +mkdir resources +kustomize build $MY_RESOURCE_DIR >resources/all.yaml +# or +helm template release-name chart-name >resources/all.yaml +# or +cp $MY_RESOURCE_DIR/*.yaml resources/ +``` All pod definitions and templates in the resources need an additional init container that talks to -the coordinator and eventually obtains a certificate to use for mesh communication between -confidential workloads. The definition can be taken directly from -[deployments/simple/initializer.yml](deployments/simple/initializer.yml). Furthermore, the runtime -class needs to be set to `kata-cc-isolation` so that the workloads are started as confidential -containers. +the coordinator. Furthermore, the runtime class needs to be set to `kata-cc-isolation` so that the +workloads are started as confidential containers. + +```yaml +spec: # v1.PodSpec + runtimeClassName: kata-cc-isolation + initContainers: + - name: initializer + image: "ghcr.io/edgelesssys/nunki/initializer:latest" + env: + - name: COORDINATOR_HOST + value: coordinator + volumeMounts: + - name: tls-certs + mountPath: /tls-config + volumes: + - name: tls-certs + emptyDir: {} +``` -Finally, you will need to deploy the Nunki coordinator, too. Copy the deployment from -[deployments/simple/coordinator.yml](deployments/simple/coordinator.yml) and adjust it as you see -fit (e.g. labels, namespace, service attributes). +Finally, you will need to deploy the Nunki coordinator, too. Start with the following definition +in `resources/coordinator.yaml` and adjust it as you see fit (e.g. labels, namespace, service attributes). + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: coordinator +spec: + selector: + matchLabels: + app.kubernetes.io/name: coordinator + replicas: 1 + template: + metadata: + labels: + app.kubernetes.io/name: coordinator + spec: + runtimeClassName: kata-cc-isolation + containers: + - name: coordinator + image: "ghcr.io/edgelesssys/nunki/coordinator:latest" +--- +apiVersion: v1 +kind: Service +metadata: + name: coordinator +spec: + ports: + - name: intercom + port: 7777 + protocol: TCP + - name: coordapi + port: 1313 + protocol: TCP + selector: + app.kubernetes.io/name: coordinator +``` ### Generate Policies and Manifest -With all your Kubernetes resource definitions in a directory, the next step is to define -*Policies* for them. - -The policy for a pod specifies what image the container runtime is allowed start, and what the -configuration for the container may or may not look like. Fortunately, this process is fully -automated given the resource definitiions. - -In your workspace directory, run `./nunki generate resources/*` to annotate all resources with a -policy. It takes some time to execute because it needs to download the desired container images for -inspection. Finally, it writes a *Manifest* file to the current directory. - +Generate the runtime policy from the resource definitions, attach it to the resources as +annotation and write the coordinator manifest. -The manifest maps each workload to the corresponding execution policy (identified by its digest). -During attestation, the coordinator ensures that the workload is governed by this policy. -Furthermore, the manifest contains the reference values for various confidential computing -attributes, such as platform versions and the Azure MAA keys. +```sh +./nunki generate -m manifest.json resources/*.yaml +``` ### Apply Resources -With the policy annotations in place, we are now ready to deploy. In your workspace directory, run -`kubectl apply -f resources/`. This will launch the coordinator and your workloads in the AKS -cluster. +Apply the resources to the cluster. Your workloads will block in the initialization phase until a +manifest is set at the coordinator. -During startup, the coordinator creates TLS certificates - a root certificate and an intermediate -certificate for signing workload certificates. The other workloads will stay in the initialization -phase until the manifest is published to the coordinator. +```sh +kubectl apply -f resources/ +``` + +### Connect to the Coordinator For the next steps, we will need to connect to the coordinator. If you did not expose the -coordinator with an external Kubernetes LoadBalancer, you can expose it locally with -`kubectl port-forward`. We're going to assume that the coordinator is exposed on `localhost:1313` -in the following sections. +coordinator with an external Kubernetes LoadBalancer, you can expose it locally: + +```sh +kubectl port-forward service/coordinator 1313:coordapi & +``` ### Set Manifest -The coordinator needs to know what workloads it is supposed to trust. This information is part of -the manifest we generated earlier. To publish the manifest, you run -`./nunki set -c localhost:1313` in your workspace directory. This command establishes an attested -TLS connection (aTLS) to the coordinator and sets the manifest. The coordinator proceeds to issue -certificates to those workloads that match a policy, and the regular containers start running. +Attest the coordinator and set the manifest. After this step, the coordinator should start issuing +TLS certs to the workloads, which should now leave the initialization phase. -### Verify the Coordinator +```sh +./nunki set -c localhost:1313 -m manifest.json +``` -Before we can share sensitive information with the workloads running in AKS, we need to verify -them. The chain of trust is: +### Verify the Coordinator -1. The coordinator verifies that the workload is running according to policy, and only issues a - certificate if that's the case. -2. We verify that the coordinator is running according to the coordinator policy, that the expected - manifest is set and that it's in possession of the intermediate certificate's private key. +An end user can verify the Nunki deployment without an explicit list of resources by calling the +coordinator's verification RPC over attested TLS. After successful validation, the output directory +`verify/` will be populated with the TLS root certificates for the configured manifest. -Thus, we know that every workload that presents a Nunki mesh certificate is trustworthy. To proceed -with verification, run `./nunki verify -c localhost 1313` in your workspace directory. This will -create a `verify/` sub-directory containing the manifest history and the TLS certificates. +```sh +./nunki verify -c localhost:1313 -o ./verify +``` ### Communicate with Workloads -Finally, we're all set. The workloads should have obtained certificates from the coordinator, and -its time to put them to use. Assuming your workload is exposed at `localhost:8443` (e.g. with -`kubectl port-forward`), you can establish a TLS connection by configuring the coordinator's mesh -root as a trusted CA certificate. For example, with `curl` you would run: +Connect to the workloads using the coordinator's mesh root as a trusted CA certificate. +For example, with `curl`: ```sh +kubectl port-forward service/$MY_SERVICE 8443:$MY_PORT & curl --cacert ./verify/mesh-root.pem https://localhost:8443 ```