diff --git a/confidential-containers/attestation.rst b/confidential-containers/attestation.rst new file mode 100644 index 000000000..a4202bbee --- /dev/null +++ b/confidential-containers/attestation.rst @@ -0,0 +1,45 @@ +.. _attestation: + +Attestation +=========== + +The NVIDIA Reference Architecture for Confidential Containers has remote attestation support for the CPU and GPU built-in. +Attestation allows a workload owner to cryptographically validate the guest Trusted Computing Base (TCB). +This process is facilitated by components inside the guest root file system. +When a secret resource is required inside the confidential guest (to decrypt a container image, or to decrypt a model, for instance), the guest components detect which CPU and GPU enclaves are in use and collect hardware evidence from them. +This evidence is sent to a remote verifier/broker, for example Trustee, which evaluates the evidence and conditionally releases secrets. + +Features that depend on secrets depend on attestation. +These features include: + +* Pulling encrypted images +* Authenticated container registry support +* Sealed secrets +* Direct workload requests for secrets + +To use these features, a remote verifier/broker, like Trustee, must first be provisioned in some trusted environment. + +Configure Remote Verifier/Broker (Trustee) +------------------------------------------ + +Follow the `upstream Trustee documentation `_ to provision a Trustee instance in a trusted environment with one adjustment. +To enable the remote NVIDIA verifier, Trustee must be configured to use the remote NVIDIA verifier, which uses the NVIDIA Remote Attestation Service (NRAS) to evaluate the evidence. This is not enabled by default. + +To enable the remote verifier, add the following lines to the Trustee configuration file:: + + [attestation_service.verifier_config.nvidia_verifier] + type = "Remote" + +If you are using the docker compose Trustee deployment, add the verifier type to ``kbs/config/as-config.json`` prior to starting Trustee. + +Following the upstream Trustee documentation, add the following annotation to the workload to point the guest components to Trustee: + +.. code-block:: yaml + + io.katacontainers.config.hypervisor.kernel_params: "agent.aa_kbc_params=cc_kbc::http://:" + +Now, the guest can be used with attestation. For more information on how to provision Trustee with resources and policies, refer to the `Trustee documentation `_. + +During attestation, the GPU will be set to ready. As such, when running a workload that does attestation, it is not necessary to set the ``nvrc.smi.srs=1`` kernel parameters. + +If attestation does not succeed, debugging is best done through the Trustee log. Debug mode can be enabled by setting the ``nvrc.smi.srs=1`` and ``RUST_LOG=debug`` kernel parameters in the Trustee environment. \ No newline at end of file diff --git a/confidential-containers/confidential-containers-deploy.rst b/confidential-containers/confidential-containers-deploy.rst new file mode 100644 index 000000000..1f1872a92 --- /dev/null +++ b/confidential-containers/confidential-containers-deploy.rst @@ -0,0 +1,504 @@ +.. _confidential-containers-deploy: + +****************************** +Deploy Confidential Containers +****************************** + +The page describes deploying the NVIDIA Confidential Container Reference Architecture on Kubernetes using Kata Containers and the NVIDIA GPU Operator. + +Before you begin, refer to the :doc:`Confidential Containers Reference Architecture ` for details on the reference architecture and the :doc:`Supported Platforms ` page for the supported platforms. + +Overview +======== + +The high-level workflow for configuring Confidential Containers is as follows: + +#. Configure the :ref:`Prerequisites `. + +#. :ref:`Label Nodes ` that you want to use with Confidential Containers. + +#. Install the :ref:`latest Kata Containers Helm chart `. + This installs the Kata Containers runtime binaries, UVM images and kernels, and TEE-specific shims (such as ``kata-qemu-nvidia-gpu-snp`` or ``kata-qemu-nvidia-gpu-tdx``) onto the cluster's worker nodes. + +#. Install the :ref:`NVIDIA GPU Operator configured for Confidential Containers `. + This installs the NVIDIA GPU Operator components that are required to deploy GPU passthrough workloads. + +After installation, you can change the :ref:`confidential computing mode ` and :ref:`run a sample GPU workload ` in a confidential container. + +This guide will configure your cluster to deploy Confidential Containers workloads. +Once configured, you can schedule workloads that request GPU resources and use the ``kata-qemu-nvidia-gpu-tdx`` or ``kata-qemu-nvidia-gpu-snp`` runtime classes for secure deployment. + +.. _coco-prerequisites: + +Prerequisites +============= + +* Use a supported platform for Confidential Containers. + For more information on machine setup, refer to :doc:`Supported Platforms `. + +* Ensure hosts are configured to enable hardware virtualization and Access Control Services (ACS). + With some AMD CPUs and BIOSes, ACS might be grouped under Advanced Error Reporting (AER). + Enabling these features is typically performed by configuring the host BIOS. +* Configured hosts to support IOMMU. + + * If the output from running ``ls /sys/kernel/iommu_groups`` includes 0, 1, and so on, then your host is configured for IOMMU. + * If the host is not configured or if you are unsure, add the ``amd_iommu=on`` Linux kernel command-line argument. For most Linux distributions, add the argument to the ``/etc/default/grub`` file, for instance:: + + ... + GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on modprobe.blacklist=nouveau" + ... + + Run ``sudo update-grub`` after making the change to configure the bootloader. Reboot the host after configuring the bootloader. + +* A Kubernetes cluster with cluster administrator privileges. + +* It is recommended that you configure your cluster's ``runtimeRequestTimeout`` in your `kubelet configuration `_ with a higher timeout value than the two minute default. + You could set this value to 20 minutes to match the default values for the other image pull timeout values in Kata Containers. + + Refer to the :ref:`Configure Image Pull Timeouts ` section on this page for more details on adjusting the image pull timeout values. + + +* Enable ``KubeletPodResourcesGet`` on your cluster. + The NVIDIA GPU runtime classes use VFIO cold-plug, which requires the Kata runtime to query Kubelet's Pod Resources API to discover allocated GPU devices during sandbox creation. + + * For Kubernetes versions older than 1.34, explicitly enable the ``KubeletPodResourcesGet`` feature gate in your Kubelet configuration. + * For Kubernetes 1.34 and later, this feature is enabled by default. + + Refer to the `Kata runtime (VFIO cold-plug) `_ section in the upstream NVIDIA GPU passthrough guide for more information. + +.. _installation-and-configuration: + +Installation and Configuration +=============================== + +.. _coco-label-nodes: + +Label Nodes +----------- + +#. Label the nodes that you want to use with Confidential Containers: + + .. code-block:: console + + $ kubectl label node nvidia.com/gpu.workload.config=vm-passthrough + +The GPU Operator uses this label to determine what software components to deploy to a node. +The ``nvidia.com/gpu.workload.config=vm-passthrough`` label specifies that the node should receive the software components to run Confidential Containers. +You can use this label on nodes for Confidential Containers workloads, and run traditional container workloads with GPU on other nodes in your cluster. + +.. tip:: + + Skip this section if you plan to use all nodes in your cluster to run Confidential Containers and instead set ``sandboxWorkloads.defaultWorkload=vm-passthrough`` when installing the GPU Operator. + +To check whether the node label has been added, run the following command: + +.. code-block:: console + + $ kubectl describe node | grep nvidia.com/gpu.workload.config + +Example output: + +.. code-block:: output + + nvidia.com/gpu.workload.config: vm-passthrough + +.. _coco-install-kata-chart: + +Install the Kata Containers Helm Chart +-------------------------------------- + +Install the ``kata-deploy`` Helm chart. +The ``kata-deploy`` chart installs all required components from the Kata Containers project including the Kata Containers runtime binary, runtime configuration, UVM kernel, and images that NVIDIA uses for Confidential Containers and native Kata containers. + +The minimum required version is 3.29.0. + +#. Get the latest version of the ``kata-deploy`` Helm chart: + + .. code-block:: console + + $ export VERSION="3.29.0" + $ export CHART="oci://ghcr.io/kata-containers/kata-deploy-charts/kata-deploy" + + +#. Install the kata-deploy Helm chart: + + .. code-block:: console + + $ helm install kata-deploy "${CHART}" \ + --namespace kata-system --create-namespace \ + --set nfd.enabled=false \ + --wait --timeout 10m \ + --version "${VERSION}" + + *Example Output* + + .. code-block:: output + + LAST DEPLOYED: Wed Apr 1 17:03:00 2026 + NAMESPACE: kata-system + STATUS: deployed + REVISION: 1 + DESCRIPTION: Install complete + TEST SUITE: None + +#. Optional: Verify that the kata-deploy pod is running: + + .. code-block:: console + + $ kubectl get pods -n kata-system | grep kata-deploy + + *Example Output* + + .. code-block:: output + + NAME READY STATUS RESTARTS AGE + kata-deploy-b2lzs 1/1 Running 0 6m37s + +#. Optional: View the pod in the kata-system namespace and ensure it is running: + + .. code-block:: console + + $ kubectl get pod,svc -n kata-system + + *Example Output*: + + .. code-block:: output + + NAME READY STATUS RESTARTS AGE + pod/kata-deploy-4f658 1/1 Running 0 21s + + Wait a few minutes for kata-deploy to create the base runtime classes. + +#. Verify that the ``kata-qemu-nvidia-gpu``, ``kata-qemu-nvidia-gpu-snp``, and ``kata-qemu-nvidia-gpu-tdx`` runtime classes are available: + + .. code-block:: console + + $ kubectl get runtimeclass | grep kata-qemu-nvidia-gpu + + *Example Output* + + .. code-block:: output + + NAME HANDLER AGE + kata-qemu-nvidia-gpu kata-qemu-nvidia-gpu 40s + kata-qemu-nvidia-gpu-snp kata-qemu-nvidia-gpu-snp 40s + kata-qemu-nvidia-gpu-tdx kata-qemu-nvidia-gpu-tdx 40s + + Several runtimes are installed by the ``kata-deploy`` chart. + The ``kata-qemu-nvidia-gpu`` runtime class is used with Kata Containers, in a non-Confidential Containers scenario. + The ``kata-qemu-nvidia-gpu-snp`` and ``kata-qemu-nvidia-gpu-tdx`` runtime classes are used to deploy Confidential Containers workloads. + + +.. _coco-install-gpu-operator: + +Install the NVIDIA GPU Operator +-------------------------------- + +Configure the GPU Operator to deploy Confidential Container components. + +#. Add and update the NVIDIA Helm repository: + + .. code-block:: console + + $ helm repo add nvidia https://helm.ngc.nvidia.com/nvidia \ + && helm repo update + +#. Install the GPU Operator. + The following configures the GPU Operator to deploy the operands that are required for Confidential Containers. + + + .. code-block:: console + + $ helm install --wait --generate-name \ + -n gpu-operator --create-namespace \ + nvidia/gpu-operator \ + --set sandboxWorkloads.enabled=true \ + --set sandboxWorkloads.mode=kata \ + --set nfd.enabled=true \ + --set nfd.nodefeaturerules=true \ + --version=v26.3.0 + + .. tip:: + + Add ``--set sandboxWorkloads.defaultWorkload=vm-passthrough`` if every worker node should deploy Confidential Containers by default. + + Refer to the :ref:`Confidential Containers Configuration Settings ` section on this page for more details on the Confidential Containers configuration options you can specify when installing the GPU Operator. + + Refer to the :ref:`Common chart customization options ` in :doc:`Installing the NVIDIA GPU Operator ` for more details on the additional general configuration options you can specify when installing the GPU Operator. + + *Example Output* + + .. code-block:: output + + NAME: gpu-operator + LAST DEPLOYED: Tue Mar 10 17:58:12 2026 + NAMESPACE: gpu-operator + STATUS: deployed + REVISION: 1 + TEST SUITE: None + +#. Verify that all GPU Operator pods, especially the Confidential Computing Manager, Sandbox Device Plugin and VFIO Manager operands, are running: + + .. code-block:: console + + $ kubectl get pods -n gpu-operator + + *Example Output*: + + .. code-block:: output + + NAME READY STATUS RESTARTS AGE + gpu-operator-1766001809-node-feature-discovery-gc-75776475sxzkp 1/1 Running 0 86s + gpu-operator-1766001809-node-feature-discovery-master-6869lxq2g 1/1 Running 0 86s + gpu-operator-1766001809-node-feature-discovery-worker-mh4cv 1/1 Running 0 86s + gpu-operator-f48fd66b-vtfrl 1/1 Running 0 86s + nvidia-cc-manager-7z74t 1/1 Running 0 61s + nvidia-kata-sandbox-device-plugin-daemonset-d5rvg 1/1 Running 0 30s + nvidia-sandbox-validator-6xnzc 1/1 Running 1 30s + nvidia-vfio-manager-h229x 1/1 Running 0 62s + + +#. Optional: If you have host access to the worker node, you can perform the following validation steps: + + a. Confirm that the host uses the vfio-pci device driver for GPUs:: + + $ lspci -nnk -d 10de: + + *Example Output*: + + .. code-block:: output + + 65:00.0 3D controller [0302]: NVIDIA Corporation xxxxxxx [xxx] [10de:xxxx] (rev xx) + Subsystem: NVIDIA Corporation xxxxxxx [xxx] [10de:xxxx] + Kernel driver in use: vfio-pci + Kernel modules: nvidiafb, nouveau + + +.. _coco-configuration-settings: + +Confidential Containers Configuration Settings +---------------------------------------------- + +The following are the available GPU Operator configuration settings to enable Confidential Containers: + +.. list-table:: + :widths: 20 50 30 + :header-rows: 1 + + * - Parameter + - Description + - Default + + * - ``sandboxWorkloads.enabled`` + - Enables sandbox workload management in the GPU Operator for virtual + machine-style workloads and related operands. + - ``false`` + + * - ``sandboxWorkloads.defaultWorkload`` + - Specifies the default type of workload for the cluster, one of ``container``, ``vm-passthrough``, or ``vm-vgpu``. + + Setting ``vm-passthrough`` or ``vm-vgpu`` can be helpful if you plan to run all or mostly virtual machines in your cluster. + - ``container`` + + * - ``sandboxWorkloads.mode`` + - Specifies the sandbox mode to use when deploying sandbox workloads. + Accepted values are ``kubevirt`` (default) and ``kata``. + - ``kubevirt`` + + * - ``sandboxDevicePlugin.env`` + - Optional list of environment variables passed to the NVIDIA Sandbox + Device Plugin pod. Each list item is an ``EnvVar`` object with required + ``name`` and optional ``value`` fields. + - ``[]`` (empty list) + +.. _coco-configuration-heterogeneous-clusters: + +Configuration for Heterogeneous Clusters +---------------------------------------- + +For heterogeneous clusters with different GPU types, you can specify an empty +``P_GPU_ALIAS`` environment variable for the sandbox device plugin by including +the following in your GPU Operator installation: +``--set sandboxDevicePlugin.env[0].name=P_GPU_ALIAS`` and +``--set sandboxDevicePlugin.env[0].value=""``. + +This causes the sandbox device plugin to create GPU model-specific resource +types, for example ``nvidia.com/GH100_H100L_94GB``. This is used instead of the +default ``pgpu`` type, which usually results in advertising a resource of type +``nvidia.com/pgpu``. +Use the exposed device resource types in pod specs by specifying respective resource limits. + +Similarly, NVSwitches are exposed as resources of type +``nvidia.com/nvswitch`` by default. You can use the ``NVSWITCH_ALIAS`` +environment variable to configure advertising behavior similar to +``P_GPU_ALIAS``. + +.. _coco-run-sample-workload: + +Run a Sample Workload +===================== + +A pod manifest for a confidential container GPU workload requires the following: + +1. Create a file, such as the following cuda-vectoradd-kata.yaml sample, specifying the kata-qemu-nvidia-gpu-snp runtime class: + + .. code-block:: yaml + + apiVersion: v1 + kind: Pod + metadata: + name: cuda-vectoradd-kata + namespace: default + annotations: + io.katacontainers.config.hypervisor.kernel_params: "nvrc.smi.srs=1" + spec: + runtimeClassName: kata-qemu-nvidia-gpu-snp + restartPolicy: Never + containers: + - name: cuda-vectoradd + image: "nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04" + resources: + limits: + nvidia.com/pgpu: "1" + memory: 16Gi + + Use the runtime class to specify that your workload should run in a + Confidential Container. + Specify either the ``kata-qemu-nvidia-gpu-snp`` for SEV-SNP or ``kata-qemu-nvidia-gpu-tdx`` for TDX, depending on your available nodes. + Refer to the :doc:`Supported Platforms ` page for more information on the supported platforms. + + In the sample above ``nvidia.com/pgpu`` is the default resource type for GPUs. + If you are deploying on a heterogeneous cluster, you may want to update the default behavior by specifying the ``P_GPU_ALIAS`` environment variable for the sandbox device plugin. + Refer to the :ref:`Configuration for Heterogeneous Clusters ` section on this page for more details. + +2. Create the pod: + + .. code-block:: console + + $ kubectl apply -f cuda-vectoradd-kata.yaml + +3. View the logs from pod after the container was started: + + .. code-block:: console + + $ kubectl logs -n default cuda-vectoradd-kata + + *Example Output* + + .. code-block:: output + + [Vector addition of 50000 elements] + Copy input data from the host memory to the CUDA device + CUDA kernel launch with 196 blocks of 256 threads + Copy output data from the CUDA device to the host memory + Test PASSED + Done + +4. Delete the pod: + + .. code-block:: console + + $ kubectl delete -f cuda-vectoradd-kata.yaml + +.. _managing-confidential-computing-mode: + +Managing the Confidential Computing Mode +========================================= + +You can set the default confidential computing mode of the NVIDIA GPUs by setting the ``ccManager.defaultMode=`` option. +The default value of ccManager.defaultMode is ``on``. +You can set this option when you install NVIDIA GPU Operator or afterward by modifying the cluster-policy instance of the ClusterPolicy object. + +When you change the mode, the manager performs the following actions: + +* Evicts the other GPU Operator operands from the node. + + However, the manager does not drain user workloads. You must make sure that no user workloads are running on the node before you change the mode. + +* Unbinds the GPU from the VFIO PCI device driver. +* Changes the mode and resets the GPU. +* Reschedules the other GPU Operator operands. + +The supported modes are: + +* ``on`` -- Enable Confidential Containers. +* ``off`` -- Disable Confidential Containers. +* ``ppcie`` -- Enable Confidential Containers with multi-node passthrough on HGX or DGX GPUs. + On the NVIDIA Hopper architecture multi-GPU passthrough uses protected PCIe (PPCIE) which claims exclusive use of the nvswitches for a single CVM. In this case, transition your relevant node(s) GPU mode to ``ppcie`` mode. The NVIDIA Blackwell architecture uses NVLink encryption which places the switches outside of the Trusted Computing Base (TCB) and so does not require a separate switch setting. +* ``devtools`` -- Development mode for software development and debugging. + +You can set a cluster-wide default mode and you can set the mode on individual nodes. +The mode that you set on a node has higher precedence than the cluster-wide default mode. + +Setting a Cluster-Wide Default Mode +------------------------------------ + +To set a cluster-wide mode, specify the ccManager.defaultMode field like the following example:: + + $ kubectl patch clusterpolicies.nvidia.com/cluster-policy \ + --type=merge \ + -p '{"spec": {"ccManager": {"defaultMode": "on"}}}' + +Setting a Node-Level Mode +-------------------------- + +To set a node-level mode, apply the ``nvidia.com/cc.mode=`` label like the following example:: + + $ kubectl label node nvidia.com/cc.mode=on --overwrite + +The mode that you set on a node has higher precedence than the cluster-wide default mode. + +Verifying a Mode Change +------------------------ + +To verify that changing the mode was successful, a cluster-wide or node-level change, view the nvidia.com/cc.mode and nvidia.com/cc.mode.state node labels:: + + $ kubectl get node -o json | \ + jq '.metadata.labels | with_entries(select(.key | startswith("nvidia.com/cc.mode")))' + +Example output when CC mode is disabled: + +.. code-block:: json + + { + "nvidia.com/cc.mode": "off", + "nvidia.com/cc.mode.state": "on" + } + +Example output when CC mode is enabled: + +.. code-block:: json + + { + "nvidia.com/cc.mode": "on", + "nvidia.com/cc.mode.state": "on" + } + +The "nvidia.com/cc.mode.state" variable is either "off" or "on", with "off" meaning that mode state transition is still ongoing and "on" meaning mode state transition completed. + +.. _configure-image-pull-timeouts: + +Configure Image Pull Timeouts +----------------------------- + +Using the guest-pull mechanism to securly manage images in your deployment scenarios means that pulling large images may take a significant amount of time and may delay container start. +This can lead to Kubelet de-allocating your pod before it transitions from the container creating to the container running state. + +It is recommended that you configure your cluster's ``runtimeRequestTimeout`` in your `kubelet configuration `_ with a higher timeout value than the two minute default. +You could set this value to 20 minutes (``20m``) to match the default values for the NVIDIA shim configurations in Kata Containers ``create_container_timeout`` and the agent's ``image_pull_timeout``. + +The NVIDIA shim configurations in Kata Containers use a default ``create_container_timeout`` of 1200 seconds (20 minutes). +This controls the time the shim allows for a container to remain in container creating state. + +If you need a timeout of more than 1200 seconds, you will also need to adjust the agent's ``image_pull_timeout`` value which controls the agent-side timeout for guest-image pull. +To do this, add the ``agent.image_pull_timeout`` kernel parameter to your shim configuration, or pass an explicit value in a pod annotation in the ``io.katacontainers.config.hypervisor.kernel_params:="..."`` annotation. + + +Next Steps +========== + +* Refer to the :doc:`Attestation ` page for more information on configuring attestation. +* Enable pod security policy with Agent Policy. + Refer to the `Kata Containers Agent Policy documentation `_ for more information. +* Additional NVIDIA Confidential Computing documentation is available at https://docs.nvidia.com/confidential-computing. +* Licensing information is available on the :doc:`Licensing ` page. \ No newline at end of file diff --git a/confidential-containers/index.rst b/confidential-containers/index.rst index cb806e094..d420f1b69 100644 --- a/confidential-containers/index.rst +++ b/confidential-containers/index.rst @@ -17,7 +17,7 @@ .. headings # #, * *, =, -, ^, " ********************************************************** -NVIDIA Confidential Containers Architecture (Early Access) +NVIDIA Confidential Containers Architecture ********************************************************** .. toctree:: @@ -26,10 +26,13 @@ NVIDIA Confidential Containers Architecture (Early Access) :titlesonly: Overview - Deploy Confidential Containers with NVIDIA GPU Operator + Supported Platforms + Deploy Confidential Containers + Attestation + Licensing -This is documentation for NVIDIA's Early Access implementation of Confidential Containers including reference architecture information and supported platforms. +This is documentation for NVIDIA's implementation of Confidential Containers including reference architecture information and supported platforms. .. grid:: 3 @@ -39,29 +42,29 @@ This is documentation for NVIDIA's Early Access implementation of Confidential C :link: overview :link-type: doc - Introduction and approach to Confidential Containers. + Start here to review the reference architecture, use cases, and software components. - .. grid-item-card:: :octicon:`project;1.5em;sd-mr-1` Architecture - :link: coco-architecture - :link-type: ref + .. grid-item-card:: :octicon:`server;1.5em;sd-mr-1` Supported Platforms + :link: supported-platforms + :link-type: doc - High-level flow and diagram for Confidential Containers architecture. + Learn about the validated hardware, OS, and component versions. - .. grid-item-card:: :octicon:`briefcase;1.5em;sd-mr-1` Use Cases - :link: coco-use-cases - :link-type: ref + .. grid-item-card:: :octicon:`rocket;1.5em;sd-mr-1` Deploy Confidential Containers + :link: confidential-containers-deploy + :link-type: doc - Regulated industries and workloads that benefit from confidential computing. + Use this page to deploy with the NVIDIA GPU Operator on Kubernetes. - .. grid-item-card:: :octicon:`package;1.5em;sd-mr-1` Components - :link: coco-supported-platforms-components - :link-type: ref + .. grid-item-card:: :octicon:`shield-check;1.5em;sd-mr-1` Attestation + :link: attestation + :link-type: doc - Key software components for confidential containers. + Learn about remote attestation, Trustee, and the NVIDIA verifier for GPU workloads. - .. grid-item-card:: :octicon:`server;1.5em;sd-mr-1` Supported Platforms - :link: coco-supported-platforms - :link-type: ref + .. grid-item-card:: :octicon:`law;1.5em;sd-mr-1` Licensing + :link: licensing + :link-type: doc - Platform and feature support scope for Early Access (EA). + Learn about the licensing information for Confidential Containers documentation. diff --git a/confidential-containers/licensing.rst b/confidential-containers/licensing.rst new file mode 100644 index 000000000..51355c570 --- /dev/null +++ b/confidential-containers/licensing.rst @@ -0,0 +1,10 @@ + +********* +Licensing +********* + +While the Confidential Containers (CoCo) Reference Architecture includes some components that are open source, the NVIDIA Confidential Computing capability is a licensed feature for production use cases. +To use these products, you must have a valid NVIDIA Confidential Computing license. +Refer to the `NVIDIA Product-Specific Terms for NVIDIA Confidential Computing `_ for more information. + + diff --git a/confidential-containers/overview.rst b/confidential-containers/overview.rst index 05d9d1331..65e752a68 100644 --- a/confidential-containers/overview.rst +++ b/confidential-containers/overview.rst @@ -1,5 +1,5 @@ .. license-header - SPDX-FileCopyrightText: Copyright (c) 2023 NVIDIA CORPORATION & AFFILIATES. All rights reserved. + SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. SPDX-License-Identifier: Apache-2.0 Licensed under the Apache License, Version 2.0 (the "License"); @@ -17,18 +17,18 @@ .. headings # #, * *, =, -, ^, " -****************************************************** -NVIDIA Confidential Containers Overview (Early Access) -****************************************************** +***************************************************** +NVIDIA Confidential Containers Reference Architecture +***************************************************** -.. admonition:: Early Access - - Confidential Containers are available as Early Access (EA) with curated platform and feature support. EA features are not supported in production and are not functionally complete. API and architectural designs are not final and may change. +This page describes NVIDIA's Reference Architecture for Confidential Containers. +It includes an overview of the architecture, a description of the components, and considerations for deploying workloads in Confidential Containers. .. _confidential-containers-overview: Overview ======== + NVIDIA GPUs power the training and deployment of Frontier Models—world-class Large Language Models (LLMs) that define the state of the art in AI reasoning and capability. As organizations adopt these models in regulated industries such as financial services, healthcare, and the public sector, protecting model intellectual property and sensitive user data becomes essential. Additionally, the model deployment landscape is evolving to include public clouds, enterprise on-premises, and edge. A zero-trust posture on cloud-native platforms such as Kubernetes is essential to secure assets (model IP and enterprise private data) from untrusted infrastructure with privileged user access. @@ -38,7 +38,23 @@ Securing data at rest and in transit is standard. Protecting data in-use remains In addition to TEEs, Confidential Computing provides Remote Attestation features. Attestation enables remote systems or users to interrogate the security state of a TEE before interacting with it and providing any secrets or sensitive data. `Confidential Containers `_ (CoCo) is the cloud-native approach of CC on Kubernetes. -The Confidential Containers architecture leverages Kata Containers to provide the sandboxing capabilities. `Kata Containers `_ is an open-source project that provides lightweight Utility Virtual Machines (UVMs) that feel and perform like containers while providing strong workload isolation. Along with the Confidential Containers project, Kata enables the orchestration of secure, GPU-accelerated workloads in Kubernetes. +The Confidential Containers project leverages Kata Containers to provide the sandboxing capabilities. `Kata Containers `_ is an open-source project that provides lightweight Utility Virtual Machines (UVMs) that feel and perform like containers while providing strong workload isolation. Along with the Confidential Containers project, Kata enables the orchestration of secure, GPU-accelerated workloads in Kubernetes. + +.. _coco-use-cases: + +Use Cases +========= + +The target for Confidential Containers is to enable model providers (Closed and Open source) and Enterprises to use the advancements of Gen AI, agnostic to the deployment model (Cloud, Enterprise, or Edge). Some of the key use cases that CC and Confidential Containers enable are: + +* **Zero-Trust AI & IP Protection:** You can deploy proprietary models (like LLMs) on third-party or private infrastructure. The model weights remain encrypted and are only decrypted inside the hardware-protected enclave, ensuring absolute IP protection from the host. +* **Data Clean Rooms:** This allows you to process sensitive enterprise data (like financial analytics or healthcare records) securely. Neither the infrastructure provider nor the model builder can see the raw data. + +.. image:: graphics/CoCo-Sample-Workflow.png + :alt: Sample Workflow for Securing Model IP on Untrusted Infrastructure with CoCo + +*Sample Workflow for Securing Model IP on Untrusted Infrastructure with CoCo* + .. _coco-architecture: @@ -48,74 +64,77 @@ Architecture Overview NVIDIA's approach to the Confidential Containers architecture delivers on the key promise of Confidential Computing: confidentiality, integrity, and verifiability. Integrating open source and NVIDIA software components with the Confidential Computing capabilities of NVIDIA GPUs, the Reference Architecture for Confidential Containers is designed to be the secure and trusted deployment model for AI workloads. -.. image:: graphics/CoCo-Reference-Architecture.png - :alt: High-Level Reference Architecture for Confidential Containers - -*High-Level Reference Architecture for Confidential Containers* - -The key value proposition for this architecture approach is: +The key values of this architecture approach are: 1. **Built on OSS standards** - The Reference Architecture for Confidential Containers is built on key OSS components such as Kata, Trustee, QEMU, OVMF, and Node Feature Discovery (NFD), along with hardened NVIDIA components like NVIDIA GPU Operator. 2. **Highest level of isolation** - The Confidential Containers architecture is built on Kata containers, which is the industry standard for providing hardened sandbox isolation, and augmenting it with support for GPU passthrough to Kata containers makes the base of the Trusted Execution Environment (TEE). 3. **Zero-trust execution with attestation** - Ensuring the trust of the model providers/data owners by providing a full-stack verification capability with attestation. The integration of NVIDIA GPU attestation capabilities with Trustee based architecture, to provide composite attestation provides the base for secure, attestation based key-release for encrypted workloads, deployed inside the TEE. -.. _coco-use-cases: - -Use Cases -========= - -The target for Confidential Containers is to enable model providers (Closed and Open source) and Enterprises to leverage the advancements of Gen AI, agnostic to the deployment model (Cloud, Enterprise, or Edge). Some of the key use cases that CC and Confidential Containers enable are: - -* **Zero-Trust AI & IP Protection:** You can deploy proprietary models (like LLMs) on third-party or private infrastructure. The model weights remain encrypted and are only decrypted inside the hardware-protected enclave, ensuring absolute IP protection from the host. -* **Data Clean Rooms:** This allows you to process sensitive enterprise data (like financial analytics or healthcare records) securely. Neither the infrastructure provider nor the model builder can see the raw data. +.. image:: graphics/CoCo-Reference-Architecture.png + :alt: High-Level Reference Architecture for Confidential Containers -.. image:: graphics/CoCo-Sample-Workflow.png - :alt: Sample Workflow for Securing Model IP on Untrusted Infrastructure with CoCo +*High-Level Reference Architecture for Confidential Containers* -*Sample Workflow for Securing Model IP on Untrusted Infrastructure with CoCo* +The above diagram shows the high-level reference architecture for Confidential Containers and +the key components that are used to deploy and manage Confidential Containers workloads. +The components are described in more detail in the next section. .. _coco-supported-platforms-components: Software Components for Confidential Containers =============================================== -The following is a brief overview of the software components for Confidential Containers. +The following is a brief overview of the software components in NVIDIA's Reference Architecture for Confidential Containers. +Refer to the diagram above for a visual representation of the components. + **Kata Containers** -Acts as the secure isolation layer by running standard Kubernetes Pods inside lightweight, hardware-isolated Utility VMs (UVMs) rather than sharing the untrusted host kernel. Kata containers are integrated with the Kubernetes `Agent Sandbox `_ project to deliver sandboxing capabilities. +Acts as the secure isolation layer by running standard Kubernetes Pods inside lightweight, hardware-isolated Utility Virtual Machines (UVMs) rather than sharing the untrusted host kernel. +`Kata Containers `_ is an open source project, which is integrated with the Kubernetes `Agent Sandbox `_ project, that delivers sandboxing capabilities. + +**Kata Deploy** + +Deployment mechanism (often managed with Helm) that installs the Kata runtime binaries, UVM images and kernels, and TEE-specific shims (such as ``kata-qemu-nvidia-gpu-snp`` or ``kata-qemu-nvidia-gpu-tdx``) onto the cluster's worker nodes. + +Refer to the `Kata Containers documentation `_ for more information. **NVIDIA GPU Operator** -Automates GPU lifecycle management. For Confidential Containers, it securely provisions GPU support and handles VFIO-based GPU passthrough directly into the Kata confidential VM without breaking the hardware trust boundary. +Automates GPU lifecycle management. +For Confidential Containers, it securely provisions GPU support and handles VFIO-based GPU passthrough directly into the Kata confidential Virtual Machine (VM) without breaking the hardware trust boundary. -The GPU Operator deploys the components needed to run Confidential Containers to simplify managing the software required for confidential computing and deploying confidential container workloads: +The GPU Operator deploys the components needed to run Confidential Containers to simplify managing the software required for confidential computing and deploying confidential container workloads. +These components include: * NVIDIA Confidential Computing Manager (cc-manager) for Kubernetes - to set the confidential computing (CC) mode on the NVIDIA GPUs. -* NVIDIA Sandbox Device Plugin - to discover NVIDIA GPUs along with their capabilities, to advertise these to Kubernetes, and to allocate GPUs during pod deployment. -* NVIDIA VFIO Manager - to bind discovered NVIDIA GPUs to the vfio-pci driver for VFIO passthrough. -* NVIDIA Kata Manager for Kubernetes - to create host-side CDI specifications for GPU passthrough. + This manager is deployed by default on all nodes that have the ``nvidia.com/gpu.workload.config=vm-passthrough`` label. +* NVIDIA Kata Sandbox Device Plugin - to create host-side Container Device Interface (CDI) specifications for GPU passthrough, to discover NVIDIA GPUs along with their capabilities, to advertise these to Kubernetes, and to allocate GPUs during pod deployment. +* NVIDIA VFIO Manager - to bind discovered NVIDIA GPUs and NVSwitches to the vfio-pci driver for VFIO passthrough. -**Kata Deploy** - -Deployment mechanism (often managed via Helm) that installs the Kata runtime binaries, UVM images and kernels, and TEE-specific shims (such as ``kata-qemu-nvidia-gpu-snp`` or ``kata-qemu-nvidia-gpu-tdx``) onto the cluster's worker nodes. +Refer to the :doc:`NVIDIA GPU Operator ` page for more information on the NVIDIA GPU Operator. **Node Feature Discovery (NFD)** -Bootstraps the node by advertising the node features via labels to make sophisticated scheduling decisions, like installing the Kata/CoCo stack only on the nodes that support the CC prerequisites for CPU and GPU. This feature directs the Operator to install node feature rules that detect CPU security features and the NVIDIA GPU hardware. +Bootstraps the node by advertising the node features using labels to make sophisticated scheduling decisions, like installing the Kata/CoCo stack only on the nodes that support the CC prerequisites for CPU and GPU. This feature directs the Operator to install node feature rules that detect CPU security features and the NVIDIA GPU hardware. -**Trustee** +Refer to the `Node Feature Discovery documentation `_ for upstream usage and reference material. +The project source repository is `kubernetes-sigs/node-feature-discovery `_ on GitHub. -Attestation and key brokering framework (which includes the Key Broker Service and Attestation Service). It acts as the cryptographic gatekeeper, verifying hardware/software evidence and only releasing secrets if the environment is proven secure. - -**Snapshotter (e.g., Nydus)** +**Snapshotter (for example, Nydus)** Handles the container image "guest pull" functionality. Used as a remote snapshotter, it bypasses image pulls on the host. Instead, it fetches and unpacks encrypted and signed container images directly inside the protected guest memory, keeping proprietary contents hidden and ensuring image integrity. + **Kata Agent and Agent Security Policy** Runs inside the guest VM to manage the container lifecycle while enforcing a strict, immutable agent security policy based on Rego (regorus). This blocks the untrusted host from executing unauthorized commands, such as a malicious ``kubectl exec``. +**Trustee and Attestation Service** + +Attestation and key brokering framework (which includes the Key Broker Service and Attestation Service). It acts as the cryptographic gatekeeper, verifying hardware/software evidence and only releasing secrets if the environment is proven secure. + + **Confidential Data Hub (CDH)** An in-guest component that securely receives sealed secrets from Trustee and transparently manages encrypted persistent storage and image decryption for the workload. @@ -124,69 +143,24 @@ An in-guest component that securely receives sealed secrets from Trustee and tra A minimal hardened init system that securely bootstraps the guest environment, life cycles the kata-agent, provides health checks on started helper daemons while drastically reducing the attack surface. -Software Stack and Component Versions --------------------------------------- -The following is the component stack to support the open Reference Architecture (RA) along with the proposed versions of different SW components. +GPU Operator Cluster Topology Considerations +-------------------------------------------- -.. flat-table:: - :header-rows: 1 +The GPU Operator deploys and manages components for allocating and utilizing the GPU resources on your cluster. +Depending on how you configure the Operator, different components are deployed on the worker nodes. +When setting up Confidential Containers support, you can configure all the worker nodes in your cluster for running GPU workloads with Confidential Containers, or you can configure some nodes for Confidential Containers and the others for traditional containers. +This configuration is done through node labelling and configuration flags set during installation or by editing the ClusterPolicy object post installation. - * - Category - - Component - - Release/Version - * - :rspan:`1` **HW Platform** - - GPU Platform - - | Hopper 100/200 - | Blackwell B200 - | Blackwell RTX Pro 6000 - * - CPU Platform - - | AMD Genoa/ Milan - | Intel ER/ GR - * - :rspan:`7` **Host SW Components** - - Host OS - - 25.10 - * - Host Kernel - - 6.17+ - * - Guest OS - - Distroless - * - Guest kernel - - 6.18.5 - * - OVMF - - edk2-stable202511 - * - QEMU - - 10.1 \+ Patches - * - Containerd - - 2.2.2 \+ - * - Kubernetes - - 1.32 \+ - * - :rspan:`3` **Confidential Containers Core Components** - - NFD - - v0.6.0 - * - NVIDIA/gpu-operator - | - NVIDIA VFIO Manager - | - NVIDIA Sandbox device plugin - | - NVIDIA Confidential Computing Manager for Kubernetes - | - NVIDIA Kata Manager for Kubernetes - - v25.10.0 and higher - * - CoCo release (EA) - | - Kata 3.25 (w/ kata-deploy helm) - | - Trustee/Guest components 0.17.0 - | - KBS protocol 0.4.0 - - v0.18.0 - - -Cluster Topology Considerations -------------------------------- - -You can configure all the worker nodes in your cluster for running GPU workloads with confidential containers, or you can configure some nodes for Confidential Containers and the others for traditional containers. Consider the following example where node A is configured to run traditional containers and node B is configured to run confidential containers. + +Consider the following example where node A is configured to run traditional containers and node B is configured to run confidential containers. .. list-table:: :widths: 50 50 :header-rows: 1 - * - Node A - Traditional Containers receives the following software components - - Node B - Kata CoCo receives the following software components + * - Node A - Traditional Container nodes receive the following software components + - Node B - Confidential Container nodes receive the following software components * - * NVIDIA Driver Manager for Kubernetes * NVIDIA Container Toolkit * NVIDIA Device Plugin for Kubernetes @@ -194,52 +168,58 @@ You can configure all the worker nodes in your cluster for running GPU workloads * NVIDIA MIG Manager for Kubernetes * Node Feature Discovery * NVIDIA GPU Feature Discovery - - * NVIDIA Kata Manager for Kubernetes - * NVIDIA Confidential Computing Manager for Kubernetes + - * NVIDIA Confidential Computing Manager for Kubernetes * NVIDIA Sandbox Device Plugin * NVIDIA VFIO Manager * Node Feature Discovery -This configuration can be controlled via node labelling, as described in the `GPU Operator confidential containers deployment guide `_. +This configuration can be controlled through node labelling, as described in the :doc:`Confidential Containers deployment guide `. -.. _coco-supported-platforms: +Supported Features and Deployment Scenarios +=========================================== -Supported Platforms -=================== +The following features are supported with Confidential Containers: -Following is the platform and feature support scope for Early Access (EA) of Confidential Containers open Reference Architecture published by NVIDIA. +* Support for Confidential Container workloads as -.. flat-table:: Supported Platforms - :header-rows: 1 + * Single‑GPU passthrough (one physical GPU per pod). + * Multi‑GPU passthrough on NVSwitch (NVLink) based HGX systems. + For example, NVIDIA HGX Hopper (SXM) and NVIDIA HGX Blackwell or NVIDIA HGX B200. +* Composite attestation using Trustee and the NVIDIA Remote Attestation Service (NRAS). +* Generating Kata Agent Security Policies using the `genpolicy tool `_. +* Use of signed sealed secrets. +* Access to authenticated registries for container image guest-pull. +* Container image signature verification and encrypted container images. +* Ephemeral container data and image layer storage. - * - Component - - Feature - * - GPU Platform - - Hopper 100/200 - * - TEE - - AMD SEV-SNP only - * - Feature Support - - Confidential Containers w/ Kata; Single GPU Passthrough only - * - Attestation Support - - Composite Attestation for CPU \+ GPU; integration with Trustee for local verifier. +Limitations and Restrictions +---------------------------- -Refer to the *Confidential Computing Deployment Guide* at the `Confidential Computing `_ website for information about supported NVIDIA GPUs, such as the NVIDIA Hopper H100, and specifically to `CC deployment guide for SEV-SNP `_ for setup specific to AMD SEV-SNP machines. +* NVIDIA supports the GPU Operator and confidential computing with the containerd runtime only. +* Image signature verification for signed multi-arch images is currently not supported. -The following topics in the deployment guide apply to a cloud-native environment: +Getting Started +=============== +Refer to the following pages to learn more about deploying with Confidential Containers: -* Hardware selection and initial hardware configuration, such as BIOS settings. -* Host operating system selection, initial configuration, and validation. +.. grid:: 3 + :gutter: 3 -When following the cloud-native sections in the deployment guide linked above, use Ubuntu 25.10 as the host OS with its default kernel version and configuration. + .. grid-item-card:: :octicon:`server;1.5em;sd-mr-1` Supported Platforms + :link: supported-platforms + :link-type: doc -The remaining configuration topics in the deployment guide do not apply to a cloud-native environment. NVIDIA GPU Operator performs the actions that are described in these topics. + Hardware, OS, and component versions validated for general availability (GA). -Limitations and Restrictions for CoCo EA ----------------------------------------- + .. grid-item-card:: :octicon:`rocket;1.5em;sd-mr-1` Deploy Confidential Containers + :link: confidential-containers-deploy + :link-type: doc + + Deploy with the NVIDIA GPU Operator on Kubernetes. + + .. grid-item-card:: :octicon:`shield-check;1.5em;sd-mr-1` Attestation + :link: attestation + :link-type: doc + + Remote attestation, Trustee, and the NVIDIA verifier for GPU workloads. -* Only the AMD platform using SEV-SNP is supported for Confidential Containers Early Access. -* GPUs are available to containers as a single GPU in passthrough mode only. Multi-GPU passthrough and vGPU are not supported. -* Support is limited to initial installation and configuration only. Upgrade and configuration of existing clusters to configure confidential computing is not supported. -* Support for confidential computing environments is limited to the implementation described on this page. -* NVIDIA supports the GPU Operator and confidential computing with the containerd runtime only. -* NFD doesn't label all Confidential Container capable nodes as such automatically. In some cases, users must manually label nodes to deploy the NVIDIA Confidential Computing Manager for Kubernetes operand onto these nodes as described in the deployment guide. diff --git a/confidential-containers/supported-platforms.rst b/confidential-containers/supported-platforms.rst new file mode 100644 index 000000000..325251d51 --- /dev/null +++ b/confidential-containers/supported-platforms.rst @@ -0,0 +1,84 @@ +Supported Platforms +==================== + +Following are the platforms supported by the NVIDIA Confidential Containers Reference Architecture. + +Supported Hardware Platform +--------------------------- + +NVIDIA GPUs +----------- + +.. flat-table:: + :header-rows: 1 + + * - GPU Platform + * - NVIDIA Hopper 100 + * - NVIDIA Hopper 200 + * - NVIDIA Blackwell B200 + * - NVIDIA Blackwell RTX Pro 6000 + +CPU Platforms +------------- + +.. flat-table:: + :header-rows: 1 + + * - Category + - Operating System + - Kernel Version + * - AMD Genoa / Milan + - Ubuntu 25.10 + - 6.17+ + * - Intel Emerald Rapids (ER) / Granite Rapids (GR) + - Ubuntu 25.10 + - 6.17+ + +For additional information on node configuration, refer to the `Confidential Computing Deployment Guide `_ for information about supported NVIDIA GPUs, such as the NVIDIA Hopper H100. + +The following topics in the deployment guide apply to a cloud-native environment: + +* Hardware selection and initial hardware configuration, such as BIOS settings. +* Host operating system selection, initial configuration, and validation. + +When following the cloud-native sections in the deployment guide linked above, use Ubuntu 25.10 as the host OS with its default kernel version and configuration. + +For additional resources on machine setup: + +* Refer to the `NVIDIA Trusted Computing Solutions website `_. +* Refer to the :doc:`Licensing ` page for more information on the licensing requirements for NVIDIA Confidential Computing capabilities. + +Supported Software Components +----------------------------- + +.. flat-table:: + :header-rows: 1 + + * - Component + - Release/Version + * - Guest OS + - Distroless + * - Guest kernel + - 6.18.5 + * - OVMF + - edk2-stable202511 + * - QEMU + - 10.1 \+ Patches + * - Containerd + - 2.2.2 \+ + * - Kubernetes + - 1.32 \+ + * - Node Feature Discovery (NFD) + - v0.6.0 + * - NVIDIA GPU Operator + - v26.3.0 and higher + * - Kata Containers + - 3.29 (w/ kata-deploy helm) + * - Key Broker Service (KBS) protocol + - 0.4.0 + + +.. _coco-supported-platforms: + + + diff --git a/gpu-operator/confidential-containers-deploy.rst b/gpu-operator/confidential-containers-deploy.rst new file mode 100644 index 000000000..38e84a692 --- /dev/null +++ b/gpu-operator/confidential-containers-deploy.rst @@ -0,0 +1,31 @@ +.. _confidential-containers-deploy: + +*********************** +Confidential Containers +*********************** + +The NVIDIA GPU Operator supports deploying Confidential Containers using Kata Containers and NVIDIA Reference Architecture for Confidential Containers. + +Documentation for configuring the GPU Operator for Confidential Containers is available in the `NVIDIA Reference Architecture for Confidential Containers documentation `_. + +.. grid:: 3 + :gutter: 3 + + .. grid-item-card:: :octicon:`book;1.5em;sd-mr-1` Reference Architecture + :link: https://docs.nvidia.com/datacenter/cloud-native/confidential-containers/latest/overview.html + :link-type: url + + Overview, reference architecture, and software components for Confidential Containers. + + .. grid-item-card:: :octicon:`rocket;1.5em;sd-mr-1` Deploy + :link: https://docs.nvidia.com/datacenter/cloud-native/confidential-containers/latest/confidential-containers-deploy.html + :link-type: url + + Deploy Confidential Containers with the NVIDIA GPU Operator on Kubernetes. + + .. grid-item-card:: :octicon:`server;1.5em;sd-mr-1` Supported Platforms + :link: https://docs.nvidia.com/datacenter/cloud-native/confidential-containers/latest/supported-platforms.html + :link-type: url + + Hardware, host, and component versions validated for general availability (GA). + diff --git a/gpu-operator/getting-started.rst b/gpu-operator/getting-started.rst index f15420d96..62ac510c4 100644 --- a/gpu-operator/getting-started.rst +++ b/gpu-operator/getting-started.rst @@ -774,4 +774,4 @@ Installation on Commercially Supported Kubernetes Platforms - |nvaie-tanzu|_ * - Google Cloud Anthos - - :external+edge:doc:`anthos-guide` + - :external+edge:doc:`anthos-guide` \ No newline at end of file diff --git a/gpu-operator/index.rst b/gpu-operator/index.rst index b3d2546c5..8d2dff342 100644 --- a/gpu-operator/index.rst +++ b/gpu-operator/index.rst @@ -56,6 +56,7 @@ :hidden: KubeVirt + Confidential Containers .. toctree:: :caption: Specialized Networks diff --git a/repo.toml b/repo.toml index b96407585..229dd7eac 100644 --- a/repo.toml +++ b/repo.toml @@ -197,8 +197,7 @@ redirects = [ { path="openshift/install-gpu-ocp.html", project="openshift", target="install-gpu-ocp.html" }, { path="dra-crds.html", target="dra-intro-install.html" }, { path="dra-gpus.html", target="dra-intro-install.html" }, - { path="confidential-containers-deploy.html", project="confidential-containers", target="overview.html" }, - { path="confidential-containers.html", project="confidential-containers", target="overview.html" }, + { path="confidential-containers.html", target="confidential-containers-deploy.html" }, ] [repo_docs.projects.gpu-operator.builds.linkcheck]