Skip to main content
Version: Next 🚧

Confidential computing setup

Confidential Containers (CoCo) runs on Kairos the same way it runs on any Kubernetes distribution. Follow the upstream installation guide, then opt workloads in by selecting the appropriate runtime class. This page covers what is specific to Kairos: the host settings SEV-SNP needs, and the changes needed on Hadron.

Host requirements​

Enable the virtualization and memory-encryption options in the firmware: SEV, SEV-SNP and SNP memory coverage on AMD, or the equivalent TDX settings on Intel.

On AMD, also check the Minimum SEV non-ES ASID setting (called SEV-ES ASID Space Limit on some boards). Only the ASIDs below that value can be used by SEV-ES and SEV-SNP guests, so a value of 1 leaves none and the kernel keeps SNP disabled. Set it to the number of SEV-ES/SNP guests you plan to run plus one. After a reboot, the kernel should report SNP as enabled:

dmesg | grep kvm_amd # SEV-SNP enabled
cat /sys/module/kvm_amd/parameters/sev_snp # Y

Ubuntu and other glibc-based images​

Kairos no longer ships prebuilt Ubuntu images, so build your own with the kernel command-line flags and vendor packages (Intel TDX/SGX attestation libraries, container image decryption tooling, etc.) your scenario needs. The upstream install then works unchanged. See:

Hadron​

Hadron is musl-only. Since Kata 4.0.0, the Kata host binaries the CoCo chart installs (containerd-shim-kata-v2, kata-runtime, kata-monitor) are built statically (kata-containers#13243), so they run on Hadron. The nydus-snapshotter the chart installs for guest image pulls is still the glibc build, so nydus-for-kata-tee.service fails with 203/EXEC and pods never get a sandbox.

Pick one of the two setups below. The examples use chart 0.23.0 (Kata 4.2.0). Add any values your Kubernetes distribution needs from the upstream installation guide.

Kata can pull images inside the guest without nydus-snapshotter, using its force guest-pull mode (marked experimental upstream). There is then no snapshotter to install and nothing on the host to patch.

Save this values file. It skips the nydus setup and moves every shim the chart wires to nydus over to force guest-pull. All the TEE shims stay enabled:

coco-hadron.yaml
kata-as-coco-runtime:
snapshotter:
setup: []
shims:
qemu-snp:
containerd:
snapshotter: null
forceGuestPull: true
qemu-tdx:
containerd:
snapshotter: null
forceGuestPull: true
qemu-coco-dev:
containerd:
snapshotter: null
forceGuestPull: true
qemu-coco-dev-runtime-rs:
containerd:
snapshotter: null
forceGuestPull: true
qemu-nvidia-gpu-snp:
containerd:
snapshotter: null
forceGuestPull: true
qemu-nvidia-gpu-snp-runtime-rs:
containerd:
snapshotter: null
forceGuestPull: true
qemu-nvidia-gpu-tdx:
containerd:
snapshotter: null
forceGuestPull: true
qemu-nvidia-gpu-tdx-runtime-rs:
containerd:
snapshotter: null
forceGuestPull: true

Install the chart and check that the runtime class exists:

helm install coco oci://ghcr.io/confidential-containers/charts/confidential-containers \
--version 0.23.0 -n coco-system --create-namespace \
-f coco-hadron.yaml --wait
kubectl -n coco-system rollout status ds/kata-as-coco-runtime
kubectl get runtimeclass kata-qemu-snp

Run a test workload​

snp-test.yaml
apiVersion: v1
kind: Pod
metadata:
name: snp-test
spec:
runtimeClassName: kata-qemu-snp
restartPolicy: Never
containers:
- name: test
image: busybox:1.36
command: ["sh", "-c", "dmesg | grep -iE 'sev|snp|memory encryption'; uname -r; sleep 3600"]
kubectl apply -f snp-test.yaml
kubectl logs snp-test

The logs come from the guest kernel and should include Memory Encryption Features active: AMD SEV SEV-ES SEV-SNP.

Legacy enclave-cc integration​

Warning

The rest of this page describes an earlier, experimental integration based on enclave-cc and the coco community bundle. It is kept for historical reference and does not reflect the current recommended approach described above.

Confidential computing is a type of secure computing that allows users to encrypt and decrypt data on a secure, isolated computing environment. It works by encrypting the data before it is sent to the cloud or other computing resources. This allows users to keep their data private and secure, even if it is accessed by unauthorized parties. This makes it useful for sensitive data such as financial information, health records, and other confidential data.

One important aspect of Confidential Computing is the ability to encrypt data even in-memory. This document describes how to setup Kairos to use enclave-cc in order to run confidential workloads.

Create a Kairos cluster​

The coco community bundle is supported since Kairos version v2.0.0-alpha3 ("coco" stands for "Confidential Computing").

A configuration file like the following should be used (see the bundles section):

#cloud-config
bundles:
- targets:
- run://quay.io/kairos/community-bundles:cert-manager_latest
- run://quay.io/kairos/community-bundles:kairos_latest
- run://quay.io/kairos/community-bundles:coco_latest

install:
auto: true
device: auto
reboot: true

k3s:
enabled: true

users:
- name: kairos
passwd: kairos
groups:
- admin

The bundle is making some changes on the host's filesystem (installs a customized containerd binary among other things) and a restart of the node is needed in order for the changes to be applied fully. When this file appears, reboot the node: /etc/containerd/.sentinel.

Additional steps​

kubectl label --overwrite node $(kubectl get nodes -o jsonpath='{.items[].metadata.name}') node-role.kubernetes.io/worker=""
kubectl apply -k github.com/confidential-containers/operator/config/release?ref=v0.4.0
  • [Deploy the ccruntime resource]
kubectl apply -k github.com/confidential-containers/operator/config/samples/ccruntime/ssh-demo?ref=v0.4.0

(wait until they are all running: kubectl get pods -n confidential-containers-system --watch)

  • Deploy a workload

    The last part with the verification will only work from within a Pod because the IP address is internal:

    ssh -i ccv0-ssh root@$(kubectl get service ccv0-ssh -o jsonpath="{.spec.clusterIP}")

    You can create a Pod like this:

    apiVersion: v1
    kind: Pod
    metadata:
    name: kubectl
    spec:
    containers:
    - name: kubectl
    image: opensuse/leap
    command: ["/bin/sh", "-ec", "trap : TERM INT; sleep infinity & wait"]

    Get a shell to it and run the verification commands (You will need to install ssh in the Pod first).

Known limitations​

The above solution has some known limitations that might be addressed in future releases of Kairos. Namely:

  • After a Kairos upgrade, the above process has to be repeated in order to install the customized containerd and the relevant configuration.
  • There is no simple way to upgrade to newer versions of the bundle (this is a general bundles limitation).