From aa0f6d54d3129d20813a35f10197a76ee15f1c24 Mon Sep 17 00:00:00 2001 From: Markus Rudy Date: Fri, 26 Jan 2024 10:35:28 +0100 Subject: [PATCH 1/2] describe user workflow in README --- README.md | 94 +++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 94 insertions(+) diff --git a/README.md b/README.md index 21a2bca8e..4775f88aa 100644 --- a/README.md +++ b/README.md @@ -27,6 +27,100 @@ app, making it easier to assure stakeholders of your app's security. Nunki provides an Initializer that handles the remote attestation on the workload side transparently and fetches the workload certificate. The Initializer runs as init container before your workload is started. +## Generic Workflow + +### Setup + +First of all, you will need a Nunki binary and a workspace directory. At this stage of the project, +you will need to follow the instructions in [CONTRIBUTING.md](CONTRIBUTING.md) to set up a +development environment and then run `just demodir` to initialize a workspace. In the future, this +step may become part of the released Nunki binary. + +### Kubernetes Resources + +In this step, you will define Kubernetes resources for the Nunki coordinator and the workloads you +want to run. Nunki works with vanilla Kubernetes yaml, so all you have to do is copy the resource +files into a `resources` sub-directory of your workspace. If you use Helm, you can do that with +`helm template`; if you use Kustomize, you can do that with `kustomize build`. + +All pod definitions and templates in the resources need an additional init container that talks to +the coordinator and eventually obtains a certificate to use for mesh communication between +confidential workloads. The definition can be taken directly from +[deployments/simple/initializer.yml](deployments/simple/initializer.yml). Furthermore, the runtime +class needs to be set to `kata-cc-isolation` so that the workloads are started as confidential +containers. + +Finally, you will need to deploy the Nunki coordinator, too. Copy the deployment from +[deployments/simple/coordinator.yml](deployments/simple/coordinator.yml) and adjust it as you see +fit (e.g. labels, namespace, service attributes). + +### Generate Policies and Manifest + +With all your Kubernetes resource definitions in a directory, the next step is to define +*Policies* for them. + +The policy for a pod specifies what image the container runtime is allowed start, and what the +configuration for the container may or may not look like. Fortunately, this process is fully +automated given the resource definitiions. + +In your workspace directory, run `./nunki generate resources/*` to annotate all resources with a +policy. It takes some time to execute because it needs to download the desired container images for +inspection. Finally, it writes a *Manifest* file to the current directory. + + +The manifest maps each workload to the corresponding execution policy (identified by its digest). +During attestation, the coordinator ensures that the workload is governed by this policy. +Furthermore, the manifest contains the reference values for various confidential computing +attributes, such as platform versions and the Azure MAA keys. + +### Apply Resources + +With the policy annotations in place, we are now ready to deploy. In your workspace directory, run +`kubectl apply -f resources/`. This will launch the coordinator and your workloads in the AKS +cluster. + +During startup, the coordinator creates TLS certificates - a root certificate and an intermediate +certificate for signing workload certificates. The other workloads will stay in the initialization +phase until the manifest is published to the coordinator. + +For the next steps, we will need to connect to the coordinator. If you did not expose the +coordinator with an external Kubernetes LoadBalancer, you can expose it locally with +`kubectl port-forward`. We're going to assume that the coordinator is exposed on `localhost:1313` +in the following sections. + +### Set Manifest + +The coordinator needs to know what workloads it is supposed to trust. This information is part of +the manifest we generated earlier. To publish the manifest, you run +`./nunki set -c localhost:1313` in your workspace directory. This command establishes an attested +TLS connection (aTLS) to the coordinator and sets the manifest. The coordinator proceeds to issue +certificates to those workloads that match a policy, and the regular containers start running. + +### Verify the Coordinator + +Before we can share sensitive information with the workloads running in AKS, we need to verify +them. The chain of trust is: + +1. The coordinator verifies that the workload is running according to policy, and only issues a + certificate if that's the case. +2. We verify that the coordinator is running according to the coordinator policy, that the expected + manifest is set and that it's in possession of the intermediate certificate's private key. + +Thus, we know that every workload that presents a Nunki mesh certificate is trustworthy. To proceed +with verification, run `./nunki verify -c localhost 1313` in your workspace directory. This will +create a `verify/` sub-directory containing the manifest history and the TLS certificates. + +### Communicate with Workloads + +Finally, we're all set. The workloads should have obtained certificates from the coordinator, and +its time to put them to use. Assuming your workload is exposed at `localhost:8443` (e.g. with +`kubectl port-forward`), you can establish a TLS connection by configuring the coordinator's mesh +root as a trusted CA certificate. For example, with `curl` you would run: + +```sh +curl --cacert ./verify/mesh-root.pem https://localhost:8443 +``` + ## Contributing See the [contributing guide](CONTRIBUTING.md). From d11f27c01ec0635b2eaa9d70e56087ca8e48c3da Mon Sep 17 00:00:00 2001 From: Markus Rudy Date: Mon, 29 Jan 2024 18:30:11 +0100 Subject: [PATCH 2/2] fixup! describe user workflow in README --- README.md | 171 +++++++++++++++++++++++++++++++++++------------------- 1 file changed, 111 insertions(+), 60 deletions(-) diff --git a/README.md b/README.md index 4775f88aa..4fa133ccb 100644 --- a/README.md +++ b/README.md @@ -29,95 +29,146 @@ fetches the workload certificate. The Initializer runs as init container before ## Generic Workflow -### Setup +### Prerequisites -First of all, you will need a Nunki binary and a workspace directory. At this stage of the project, -you will need to follow the instructions in [CONTRIBUTING.md](CONTRIBUTING.md) to set up a -development environment and then run `just demodir` to initialize a workspace. In the future, this -step may become part of the released Nunki binary. +* An empty workspace directory. +* The `nunki` binary. + + +```sh +WORKSPACE=$(mktemp -d) +cd "$WORKSPACE" +curl -Lo nunki https://... +``` ### Kubernetes Resources -In this step, you will define Kubernetes resources for the Nunki coordinator and the workloads you -want to run. Nunki works with vanilla Kubernetes yaml, so all you have to do is copy the resource -files into a `resources` sub-directory of your workspace. If you use Helm, you can do that with -`helm template`; if you use Kustomize, you can do that with `kustomize build`. +All Kubernetes resources that should be running in confidential containers must be present in the +resources directory. You can generate them from a Helm chart or from a Kustomization, or just copy +them over from your repository. + +```sh +mkdir resources +kustomize build $MY_RESOURCE_DIR >resources/all.yaml +# or +helm template release-name chart-name >resources/all.yaml +# or +cp $MY_RESOURCE_DIR/*.yaml resources/ +``` All pod definitions and templates in the resources need an additional init container that talks to -the coordinator and eventually obtains a certificate to use for mesh communication between -confidential workloads. The definition can be taken directly from -[deployments/simple/initializer.yml](deployments/simple/initializer.yml). Furthermore, the runtime -class needs to be set to `kata-cc-isolation` so that the workloads are started as confidential -containers. +the coordinator. Furthermore, the runtime class needs to be set to `kata-cc-isolation` so that the +workloads are started as confidential containers. + +```yaml +spec: # v1.PodSpec + runtimeClassName: kata-cc-isolation + initContainers: + - name: initializer + image: "ghcr.io/edgelesssys/nunki/initializer:latest" + env: + - name: COORDINATOR_HOST + value: coordinator + volumeMounts: + - name: tls-certs + mountPath: /tls-config + volumes: + - name: tls-certs + emptyDir: {} +``` -Finally, you will need to deploy the Nunki coordinator, too. Copy the deployment from -[deployments/simple/coordinator.yml](deployments/simple/coordinator.yml) and adjust it as you see -fit (e.g. labels, namespace, service attributes). +Finally, you will need to deploy the Nunki coordinator, too. Start with the following definition +in `resources/coordinator.yaml` and adjust it as you see fit (e.g. labels, namespace, service attributes). + +```yaml +apiVersion: apps/v1 +kind: Deployment +metadata: + name: coordinator +spec: + selector: + matchLabels: + app.kubernetes.io/name: coordinator + replicas: 1 + template: + metadata: + labels: + app.kubernetes.io/name: coordinator + spec: + runtimeClassName: kata-cc-isolation + containers: + - name: coordinator + image: "ghcr.io/edgelesssys/nunki/coordinator:latest" +--- +apiVersion: v1 +kind: Service +metadata: + name: coordinator +spec: + ports: + - name: intercom + port: 7777 + protocol: TCP + - name: coordapi + port: 1313 + protocol: TCP + selector: + app.kubernetes.io/name: coordinator +``` ### Generate Policies and Manifest -With all your Kubernetes resource definitions in a directory, the next step is to define -*Policies* for them. - -The policy for a pod specifies what image the container runtime is allowed start, and what the -configuration for the container may or may not look like. Fortunately, this process is fully -automated given the resource definitiions. - -In your workspace directory, run `./nunki generate resources/*` to annotate all resources with a -policy. It takes some time to execute because it needs to download the desired container images for -inspection. Finally, it writes a *Manifest* file to the current directory. - +Generate the runtime policy from the resource definitions, attach it to the resources as +annotation and write the coordinator manifest. -The manifest maps each workload to the corresponding execution policy (identified by its digest). -During attestation, the coordinator ensures that the workload is governed by this policy. -Furthermore, the manifest contains the reference values for various confidential computing -attributes, such as platform versions and the Azure MAA keys. +```sh +./nunki generate -m manifest.json resources/*.yaml +``` ### Apply Resources -With the policy annotations in place, we are now ready to deploy. In your workspace directory, run -`kubectl apply -f resources/`. This will launch the coordinator and your workloads in the AKS -cluster. +Apply the resources to the cluster. Your workloads will block in the initialization phase until a +manifest is set at the coordinator. -During startup, the coordinator creates TLS certificates - a root certificate and an intermediate -certificate for signing workload certificates. The other workloads will stay in the initialization -phase until the manifest is published to the coordinator. +```sh +kubectl apply -f resources/ +``` + +### Connect to the Coordinator For the next steps, we will need to connect to the coordinator. If you did not expose the -coordinator with an external Kubernetes LoadBalancer, you can expose it locally with -`kubectl port-forward`. We're going to assume that the coordinator is exposed on `localhost:1313` -in the following sections. +coordinator with an external Kubernetes LoadBalancer, you can expose it locally: + +```sh +kubectl port-forward service/coordinator 1313:coordapi & +``` ### Set Manifest -The coordinator needs to know what workloads it is supposed to trust. This information is part of -the manifest we generated earlier. To publish the manifest, you run -`./nunki set -c localhost:1313` in your workspace directory. This command establishes an attested -TLS connection (aTLS) to the coordinator and sets the manifest. The coordinator proceeds to issue -certificates to those workloads that match a policy, and the regular containers start running. +Attest the coordinator and set the manifest. After this step, the coordinator should start issuing +TLS certs to the workloads, which should now leave the initialization phase. -### Verify the Coordinator +```sh +./nunki set -c localhost:1313 -m manifest.json +``` -Before we can share sensitive information with the workloads running in AKS, we need to verify -them. The chain of trust is: +### Verify the Coordinator -1. The coordinator verifies that the workload is running according to policy, and only issues a - certificate if that's the case. -2. We verify that the coordinator is running according to the coordinator policy, that the expected - manifest is set and that it's in possession of the intermediate certificate's private key. +An end user can verify the Nunki deployment without an explicit list of resources by calling the +coordinator's verification RPC over attested TLS. After successful validation, the output directory +`verify/` will be populated with the TLS root certificates for the configured manifest. -Thus, we know that every workload that presents a Nunki mesh certificate is trustworthy. To proceed -with verification, run `./nunki verify -c localhost 1313` in your workspace directory. This will -create a `verify/` sub-directory containing the manifest history and the TLS certificates. +```sh +./nunki verify -c localhost:1313 -o ./verify +``` ### Communicate with Workloads -Finally, we're all set. The workloads should have obtained certificates from the coordinator, and -its time to put them to use. Assuming your workload is exposed at `localhost:8443` (e.g. with -`kubectl port-forward`), you can establish a TLS connection by configuring the coordinator's mesh -root as a trusted CA certificate. For example, with `curl` you would run: +Connect to the workloads using the coordinator's mesh root as a trusted CA certificate. +For example, with `curl`: ```sh +kubectl port-forward service/$MY_SERVICE 8443:$MY_PORT & curl --cacert ./verify/mesh-root.pem https://localhost:8443 ```