Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs/deployment: add generic deployment workflow #332

Merged
merged 1 commit into from
Apr 17, 2024
Merged

docs/deployment: add generic deployment workflow #332

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
195 changes: 0 additions & 195 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -111,201 +111,6 @@ by the `node-installer` DaemonSet.
This runtime consists of a containerd runtime plugin, a virtual machine manager (cloud-hypervisor), and a podvm image (IGVM and rootFS).
The installer takes care of provisioning every node in the cluster so it provides this runtime class.

## Installation

Download the latest CLI from our release and put it into your PATH:

```sh
curl -fLo contrast https://github.com/edgelesssys/contrast/releases/download/latest/contrast
mv contrast /usr/local/bin/contrast
```

## Generic Workflow

The following instructions will guide you through the process of making an existing Kubernetes deployment
confidential and deploying it together with Contrast.

### Prerequisite

A CoCo-enabled cluster is required to run Contrast. Create it using the [`az`](https://docs.microsoft.com/en-us/cli/azure/) CLI:

```sh
# Ensure you set this to an existing resource group in your subscription
azResourceGroup="ContrastDemo"
# Select the name for your AKS cluster
azClusterName="ContrastDemo"

az extension add \
--name aks-preview \
--allow-preview true

az extension update \
--name aks-preview \
--allow-preview true

az feature register --namespace "Microsoft.ContainerService" --name "KataCcIsolationPreview"
az feature show --namespace "Microsoft.ContainerService" --name "KataCcIsolationPreview"
az provider register -n Microsoft.ContainerService

az aks create \
--resource-group "$azResourceGroup" \
--name "$azClusterName" \
--kubernetes-version 1.29 \
--os-sku AzureLinux \
--node-vm-size Standard_DC4as_cc_v5 \
--node-count 1 \
--generate-ssh-keys

az aks nodepool add \
--resource-group "$azResourceGroup" \
--name nodepool2 \
--cluster-name "$azClusterName" \
--mode System \
--node-count 1 \
--os-sku AzureLinux \
--node-vm-size Standard_DC4as_cc_v5 \
--workload-runtime KataCcIsolation

az aks get-credentials \
--resource-group "$azResourceGroup" \
--name "$azClusterName"
```

Check [Azure's deployment guide](https://learn.microsoft.com/en-us/azure/aks/deploy-confidential-containers-default-policy) for more detailed instructions.

### Deploy the Contrast Coordinator

Install the latest Contrast Coordinator release, comprising a single replica deployment and a
LoadBalancer service, into your cluster.

```sh
kubectl apply -f https://github.com/edgelesssys/contrast/releases/download/latest/coordinator.yml
```

### Prepare your Kubernetes resources

Contrast will add annotations to your Kubernetes YAML files. If you want to keep the original files
unchanged, you can copy the files into a separate local directory.
You can also generate files from a Helm chart or from a Kustomization.

```sh
mkdir resources
kustomize build $MY_RESOURCE_DIR > resources/all.yml
```

or

```sh
mkdir resources
helm template release-name chart-name > resources/all.yml
```

To specify that a workload (pod, deployment, etc.) should be deployed as confidential containers,
add `runtimeClassName: kata-cc-isolation` to the pod spec (pod definition or template).
In addition, add the Contrast Initializer as `initContainers` to these workloads and configure the
workload to use the certificates written to a `volumeMount` named `tls-certs`.

```yaml
spec: # v1.PodSpec
runtimeClassName: kata-cc-isolation
initContainers:
- name: initializer
image: "ghcr.io/edgelesssys/contrast/initializer:latest"
env:
- name: COORDINATOR_HOST
value: coordinator
volumeMounts:
- name: tls-certs
mountPath: /tls-config
volumes:
- name: tls-certs
emptyDir: {}
```

### Generate policy annotations and manifest

Run the `generate` command generate the execution policies and add them as annotations to your
deployment files. A `manifest.json` with the reference values of your deployment will be created.

```sh
./contrast generate resources/*.yml
```

### Apply Resources

Apply the resources to the cluster. Your workloads will block in the initialization phase until a
manifest is set at the Coordinator.

```sh
kubectl apply -f resources/
```

### Connect to the Contrast Coordinator

For the next steps, we will need to connect to the Coordinator. The released Coordinator resource
includes a LoadBalancer definition we can use.

```sh
coordinator=$(kubectl get svc coordinator -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')
```

> [!NOTE]
> `kubectl port-forward` uses a CRI method that isn't supported by the Kata shim. If you
> can't use a public load balancer, you can deploy a [deployments/simple/portforwarder.yml] and
> expose that with `kubectl port-forward` instead.
>
> Tracking issue: <https://github.com/kata-containers/kata-containers/issues/1693>.

### Set Manifest

Attest the Coordinator and set the manifest:

```sh
./contrast set -c "${coordinator}:1313" -m manifest.json resources/
```

After this step, the Coordinator will start issuing TLS certs to the workloads. The init container
will fetch a certificate for the workload and the workload is started.

### Verify the Coordinator

An end user (data owner) can verify the Contrast deployment using the `verify` command.

```sh
./contrast verify -c "${coordinator}:1313"
```

The CLI will attest the Coordinator using embedded reference values. The CLI will write the service mesh
root certificate and the history of manifests into the `verify/` directory. In addition, the policies referenced
in the manifest are also written to the directory.

### Communicate with Workloads

You can securely connect to the workloads using the Coordinator's `mesh-root.pem` as a trusted CA certificate.
First, expose the service on a public IP address via a LoadBalancer service:

```sh
kubectl patch svc ${MY_SERVICE} -p '{"spec": {"type": "LoadBalancer"}}'
timeout 30s bash -c 'until kubectl get service/${MY_SERVICE} --output=jsonpath='{.status.loadBalancer}' | grep "ingress"; do sleep 2 ; done'
lbip=$(kubectl get svc ${MY_SERVICE} -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo $lbip
```

Note: All workload certificates are created with a wildcard DNS entry. Since we're accessing the load balancer via IP, the SAN checks the certificate for IP entries in the SAN field. Since the certificate doesn't contain any IP entries as SAN, the validation fails.
Hence, with curl you need to skip the validation:

```sh
curl -k "https://${lbip}:443"
```

To validate the certificate with the `mesh-root.pem` locally, use `openssl` instead:

```sh
openssl s_client -showcerts -connect ${lbip}:443 </dev/null | sed -n -e '/-.BEGIN/,/-.END/ p' > certChain.pem
awk 'BEGIN {c=0;} /BEGIN CERT/{c++} { print > "cert." c ".pem"}' < certChain.pem
openssl verify -verbose -trusted verify/mesh-root.pem -- cert.1.pem
```

## Current limitations

Contrast is in an early preview stage, and most underlying projects are still under development as well.
Expand Down
161 changes: 161 additions & 0 deletions docs/docs/deployment.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,161 @@
# Workload deployment

The following instructions will guide you through the process of making an existing Kubernetes deployment
confidential and deploying it together with Contrast.

A running CoCo-enabled cluster is required for these steps, see the [setup guide](./getting-started/cluster-setup.md) on how to set it up.

## Deploy the Contrast Coordinator

Install the latest Contrast Coordinator release, comprising a single replica deployment and a
LoadBalancer service, into your cluster.

```sh
kubectl apply -f https://github.com/edgelesssys/contrast/releases/download/latest/coordinator.yml
```

## Prepare your Kubernetes resources

Contrast will add annotations to your Kubernetes YAML files. If you want to keep the original files
unchanged, you can copy the files into a separate local directory.
You can also generate files from a Helm chart or from a Kustomization.

<Tabs groupId="yaml-source">
<TabItem value="kustomize" label="kustomize">

```sh
mkdir resources
kustomize build $MY_RESOURCE_DIR > resources/all.yml
```

</TabItem>
<TabItem value="helm" label="helm">

```sh
mkdir resources
helm template $RELEASE_NAME $CHART_NAME > resources/all.yml
```

</TabItem>
<TabItem value="copy" label="copy">

```sh
cp -R $MY_RESOURCE_DIR resources/
```

</TabItem>
</Tabs>

To specify that a workload (pod, deployment, etc.) should be deployed as confidential containers,
add `runtimeClassName: kata-cc-isolation` to the pod spec (pod definition or template).
In addition, add the Contrast Initializer as `initContainers` to these workloads and configure the
workload to use the certificates written to a `volumeMount` named `tls-certs`.

```yaml
spec: # v1.PodSpec
runtimeClassName: kata-cc-isolation
initContainers:
- name: initializer
image: "ghcr.io/edgelesssys/contrast/initializer:latest"
env:
- name: COORDINATOR_HOST
value: coordinator
volumeMounts:
- name: tls-certs
mountPath: /tls-config
volumes:
- name: tls-certs
emptyDir: {}
```

## Generate policy annotations and manifest

Run the `generate` command to generate the execution policies and add them as annotations to your
deployment files. A `manifest.json` with the reference values of your deployment will be created.

```sh
contrast generate resources/
```

## Apply the resources

Apply the resources to the cluster. Your workloads will block in the initialization phase until a
manifest is set at the Coordinator.

```sh
kubectl apply -f resources/
```

## Connect to the Contrast Coordinator

For the next steps, we will need to connect to the Coordinator. The released Coordinator resource
includes a LoadBalancer definition we can use.

```sh
coordinator=$(kubectl get svc coordinator -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')
```

:::info[Port-forwarding of Confidential Containers]

`kubectl port-forward` uses a Container Runtime Interface (CRI) method that isn't supported by the Kata shim.
If you can't use a public load balancer, you can deploy a [port-forwarder](https://github.com/edgelesssys/contrast/blob/main/deployments/emojivoto/portforwarder.yml).
The port-forwarder relays traffic from a CoCo pod and can be accessed via `kubectl port-forward`.

Upstream tracking issue: https://github.com/kata-containers/kata-containers/issues/1693.

:::

## Set the manifest

Attest the Coordinator and set the manifest:

```sh
contrast set -c "${coordinator}:1313" resources/
```

After this step, the Coordinator will start issuing TLS certs to the workloads. The init container
will fetch a certificate for the workload and the workload is started.

## Verify the Coordinator

An end user (data owner) can verify the Contrast deployment using the `verify` command.

```sh
contrast verify -c "${coordinator}:1313"
```

The CLI will attest the Coordinator using embedded reference values. The CLI will write the service mesh
root certificate and the history of manifests into the `verify/` directory. In addition, the policies referenced
in the manifest are also written to the directory.

## Communicate with workloads

You can securely connect to the workloads using the Coordinator's `mesh-root.pem` as a trusted CA certificate.
First, expose the service on a public IP address via a LoadBalancer service:

```sh
kubectl patch svc ${MY_SERVICE} -p '{"spec": {"type": "LoadBalancer"}}'
timeout 30s bash -c 'until kubectl get service/${MY_SERVICE} --output=jsonpath='{.status.loadBalancer}' | grep "ingress"; do sleep 2 ; done'
lbip=$(kubectl get svc ${MY_SERVICE} -o=jsonpath='{.status.loadBalancer.ingress[0].ip}')
echo $lbip
```

:::info[Subject alternative names and LoadBalancer IP]

By default, mesh certificates are issued with a wildcard DNS entry. The web frontend is accessed
via load balancer IP in this demo. Tools like curl check the certificate for IP entries in the SAN field.
Validation fails since the certificate contains no IP entries as a subject alternative name (SAN).
For example, a connection attempt using the curl and the mesh root certificate with throw the following error:

```sh
$ curl --cacert ./verify/mesh-root.pem "https://${frontendIP}:443"
curl: (60) SSL: no alternative certificate subject name matches target host name '203.0.113.34'
```

:::

Using `openssl`, the certificate of the service can be validated with the `mesh-root.pem`:

```sh
openssl s_client -CAfile verify/mesh-root.pem -verify_return_error -connect ${frontendIP}:443 < /dev/null
```
2 changes: 1 addition & 1 deletion docs/docs/examples/emojivoto.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ openssl s_client -CAfile verify/mesh-root.pem -verify_return_error -connect ${fr

By default, mesh certificates are issued with a wildcard DNS entry. The web frontend is accessed
via load balancer IP in this demo. Tools like curl check the certificate for IP entries in the SAN field.
Validation fails since the certificate doesn't contain any IP entries as a subject alternative name (SAN).
Validation fails since the certificate contains no IP entries as a subject alternative name (SAN).
For example, a connection attempt using the curl and the mesh root certificate with throw the following error:

```sh
Expand Down
5 changes: 5 additions & 0 deletions docs/sidebars.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,11 @@ const sidebars = {
},
]
},
{
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure if this should be its own top-level section.
We'll probably add more operations that are similar to this.
For example, updating a deployment etc.
Should we have a "Workflow" section similar to Constellation or split the "Workflow" section into multiple logical groups?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can add a section for it as soon as we have a second page to go into that section? A section with a single page doesn't make much sense imo.
I'm not sure about splitting the workflow into multiple documents at this point, discussed that with @malt3 yesterday. On the one hand, we want to have a concise description of what to do to get your workflows running with Contrast, consisting of the required steps. On the other hand, we might want to give deeper background on single steps and present different alternatives or ways of doing things, leading to a format where reading is more focused than doing.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay let's add sections on the fly and keep one workflow section that includes entire procedures such as an update or a recovery.

type: 'doc',
label: 'Workload deployment',
id: 'deployment',
},
{
type: 'category',
label: 'Architecture',
Expand Down
Loading