[Bug]: Container loses access to GPU with error: "Failed to initiailize NVML: Unknown Error"

[jasonbeach]

Describe the bug
After a container has been running awhile, access to the GPU appears to be lost. Running nvidia-smi inside the container produces the "Failed to initiailize NVML: Unknown Error" message. Running nvidia-smi outside the container works fine. Rebooting laptop temporarily corrects the issue. It seems similar to issue #1133, but running sudo systemctl daemon-reload doesn't trigger the behavior. All of the software versions (listed below) are considerably newer than in that post. Starting a new container with GPU access also works. It's just long running existing containers that fail.

To Reproduce
Start a persistent container with access to GPU. Verify nvidia-smi works. Wait a while (12-24 hours). at some point access to GPU is lost and nvidia-smi no longer works.

Expected behavior
GPU access maintained.

Environment (please provide the following information):

• nvidia-container-toolkit version: 1.18.1-1
• NVIDIA Driver Version: 580.126.20
• Host OS: Ubuntu 22.04
• Kernel Version: 6.17.0-1011-oem
• Container Runtime Version: 2.2.1
• CPU Architecture: x86_64
• GPU Model(s): RTX A2000 8GB
• CUDA Version: 13.0

$ cat /etc/nvidia-container-runtime/config.toml
#accept-nvidia-visible-devices-as-volume-mounts = false
#accept-nvidia-visible-devices-envvar-when-unprivileged = true
disable-require = false
supported-driver-capabilities = "compat32,compute,display,graphics,ngx,utility,video"
#swarm-resource = "DOCKER_RESOURCE_GPU"

[nvidia-container-cli]
#debug = "/var/log/nvidia-container-toolkit.log"
environment = []
#ldcache = "/etc/ld.so.cache"
ldconfig = "@/sbin/ldconfig.real"
load-kmods = true
#no-cgroups = false
#path = "/usr/bin/nvidia-container-cli"
#root = "/run/nvidia/driver"
#user = "root:video"

[nvidia-container-runtime]
#debug = "/var/log/nvidia-container-runtime.log"
log-level = "info"
mode = "auto"
runtimes = ["runc", "crun"]

[nvidia-container-runtime.modes]

[nvidia-container-runtime.modes.cdi]
annotation-prefixes = ["cdi.k8s.io/"]
default-kind = "nvidia.com/gpu"
spec-dirs = ["/etc/cdi", "/var/run/cdi"]

[nvidia-container-runtime.modes.csv]
mount-spec-path = "/etc/nvidia-container-runtime/host-files-for-container.d"

[nvidia-container-runtime.modes.legacy]
cuda-compat-mode = "ldconfig"

[nvidia-container-runtime-hook]
path = "nvidia-container-runtime-hook"
skip-mode-detection = false

[nvidia-ctk]
path = "nvidia-ctk"

# outside of container
$ nvidia-smi
Tue Mar 24 10:10:16 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.126.20             Driver Version: 580.126.20     CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA RTX A2000 8GB Lap...    On  |   00000000:01:00.0  On |                  N/A |
| N/A   48C    P8              5W /   60W |     217MiB /   8192MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A           12581      G   /usr/lib/xorg/Xorg                       84MiB |
|    0   N/A  N/A           12992      G   /usr/bin/gnome-shell                     78MiB |
+-----------------------------------------------------------------------------------------+

[cdesiniotis]

@jasonbeach can you share the command you are using to launch your containers? If you are using docker, what is the version of docker used?

[jasonbeach]

I'm using Docker version 29.1.3, build f52814d. I believe the docker command was:

  docker run -it --runtime=nvidia -d --restart unless-stopped \
    --hostname=vision_dev \
    --name=vision_dev \
    --device /dev/video0:/dev/video0 \
    --device /dev/nvidia0  \
    --group-add video \
    --group-add dialout \
    --env SSH_PORT=8022 \
    -v /tmp/.X11-unix:/tmp/.X11-unix:rw \
    --env="DISPLAY" \
    --cap-add=SYS_PTRACE --security-opt seccomp=unconfined \
    -p 8022:8022 \
    vision_dev_uid

The actual command is generated by a script--is there a specific option you're looking for? There are a bunch of additional bind mounts that are added, but snipped here for brevity.
Thanks

[cdesiniotis]

I am interested in the options which inject GPU support into your container. You are using --runtime=nvidia. I assume your image also has the NVIDIA_VISIBLE_DEVICES environment variable set requesting one or more GPUs?

Is systemd the cgroup driver configured with docker?

[cdesiniotis]

Note, the default nvidia-container-runtime installed as part of the NVIDIA Container Toolkit v1.18.1 should not be affected by #1133. Since you mentioned that running sudo systemctl daemon-reload does not trigger this behavior, it is possible you are hitting a different issue. Do you have any insights on the events happening around the time that your containers lose access to GPUs?

[ToviHe]

Maybe you need to take a look at the DeviceAllow.conf and DevicePoilcy.conf corresponding to the container?
like 50-DeviceAllow.conf and 50-DevicePolicy.conf in /run/systemd/transient/cri-containerd-${containerd_id}.scope.d

[jasonbeach]

yes, NVIDIA_VISIBLE_DEVICES=all is defined in the container. we used to use --gpus=all to enable the gpu in our containers, but it seems in recent versions of the toolkit that didn't fully work and it was necessary to use --runtime=nvidia instead. I will note that in general we have a non-sudo user added to our container and then drop into that user as the last stage of building the image. Not sure if that makes a difference (if I recall right that's why we had to add the --device=/dev/nvidia0) so the gpu was accessible to the non-sudo user.)

$ docker info | grep 'Cgroup Driver'
 Cgroup Driver: systemd

I agree this is a separate issue from #1133.

$ cat /run/systemd/transient/docker-a499a74aa7d90bf9d81956953c9c001fcf89d5cbcfd13af0a11db4facbcd0a01.scope.d/50-DeviceAllow.conf
# This is a drop-in unit file extension, created via "systemctl set-property"
# or an equivalent operation. Do not edit.
[Scope]
DeviceAllow=
DeviceAllow=/dev/char/195:0 rwm
DeviceAllow=char-136 rwm
DeviceAllow=/dev/char/81:0 rwm
DeviceAllow=/dev/char/10:200 rwm
DeviceAllow=/dev/char/5:2 rwm
DeviceAllow=/dev/char/5:1 rwm
DeviceAllow=/dev/char/5:0 rwm
DeviceAllow=/dev/char/1:9 rwm
DeviceAllow=/dev/char/1:8 rwm
DeviceAllow=/dev/char/1:7 rwm
DeviceAllow=/dev/char/1:5 rwm
DeviceAllow=/dev/char/1:3 rwm
DeviceAllow=char-* m
DeviceAllow=block-* m

$ cat /run/systemd/transient/docker-a499a74aa7d90bf9d81956953c9c001fcf89d5cbcfd13af0a11db4facbcd0a01.scope.d/50-DevicePolicy.conf
# This is a drop-in unit file extension, created via "systemctl set-property"
# or an equivalent operation. Do not edit.
[Scope]
DevicePolicy=strict

(the folder paths were slightly different than given above.)

Unfortunately there's nothing obvious that occurs that triggers this. It's entirely possible that IT is pushing updates to the machine or running some kind of scan, but I have no visibility into that.

[cdesiniotis]

In your 50-DeviceAllow.conf I only see /dev/char/195:0 which I assume corresponds to /dev/nvidia0. However, I do not see any other entries for nvidia device nodes. Namely, I don't see /dev/char/195:255 which typically corresponds to /dev/nvidiactl.

Could you provide more information on how you installed the driver on the host? Did you install the open-source or proprietary kernel modules? Can you run ls -ltr /dev/nvidia* and /dev/char/195:* on the host?

My understanding is that access to /dev/nvidiactl is required. Below is a quick exercise that demonstrates this. I launched a container requesting one GPU:

$ docker run --rm -it --runtime=nvidia --env NVIDIA_VISIBLE_DEVICES=0 ubuntu:22.04
root@f9508d05acce:/# nvidia-smi -L
GPU 0: NVIDIA A10 (UUID: GPU-ebd34bdf-1083-eaac-2aff-4b71a022f9bd)

This is the device allow list created for this container:

$ cat /run/systemd/transient/docker-f9508d05accea776c34d6331ca1497bced1c8dabd2ad83ec6d9fc3202d0af16e.scope.d/50-DeviceAllow.conf
# This is a drop-in unit file extension, created via "systemctl set-property"
# or an equivalent operation. Do not edit.
[Scope]
DeviceAllow=
DeviceAllow=/dev/char/511:1 rwm
DeviceAllow=/dev/char/511:0 rwm
DeviceAllow=/dev/char/226:128 rwm
DeviceAllow=/dev/char/226:1 rwm
DeviceAllow=/dev/char/195:255 rwm
DeviceAllow=/dev/char/195:254 rwm
DeviceAllow=/dev/char/195:0 rwm
DeviceAllow=char-136 rwm
DeviceAllow=/dev/char/10:200 rwm
DeviceAllow=/dev/char/5:2 rwm
DeviceAllow=/dev/char/5:1 rwm
DeviceAllow=/dev/char/5:0 rwm
DeviceAllow=/dev/char/1:9 rwm
DeviceAllow=/dev/char/1:8 rwm
DeviceAllow=/dev/char/1:7 rwm
DeviceAllow=/dev/char/1:5 rwm
DeviceAllow=/dev/char/1:3 rwm
DeviceAllow=char-* m
DeviceAllow=block-* m

I then manually deleted line DeviceAllow=/dev/char/195:255 rwm from this file and ran systemd daemon-reload to force systemd to re-evaluate the cgroup's that it manages. My container then failed to execute nvidia-smi:

root@f9508d05acce:/# nvidia-smi -L
Failed to initialize NVML: Unknown Error

[ToviHe]

yes, NVIDIA_VISIBLE_DEVICES=all is defined in the container. we used to use --gpus=all to enable the gpu in our containers, but it seems in recent versions of the toolkit that didn't fully work and it was necessary to use --runtime=nvidia instead. I will note that in general we have a non-sudo user added to our container and then drop into that user as the last stage of building the image. Not sure if that makes a difference (if I recall right that's why we had to add the --device=/dev/nvidia0) so the gpu was accessible to the non-sudo user.)

$ docker info | grep 'Cgroup Driver'
 Cgroup Driver: systemd

I agree this is a separate issue from #1133.

$ cat /run/systemd/transient/docker-a499a74aa7d90bf9d81956953c9c001fcf89d5cbcfd13af0a11db4facbcd0a01.scope.d/50-DeviceAllow.conf
# This is a drop-in unit file extension, created via "systemctl set-property"
# or an equivalent operation. Do not edit.
[Scope]
DeviceAllow=
DeviceAllow=/dev/char/195:0 rwm
DeviceAllow=char-136 rwm
DeviceAllow=/dev/char/81:0 rwm
DeviceAllow=/dev/char/10:200 rwm
DeviceAllow=/dev/char/5:2 rwm
DeviceAllow=/dev/char/5:1 rwm
DeviceAllow=/dev/char/5:0 rwm
DeviceAllow=/dev/char/1:9 rwm
DeviceAllow=/dev/char/1:8 rwm
DeviceAllow=/dev/char/1:7 rwm
DeviceAllow=/dev/char/1:5 rwm
DeviceAllow=/dev/char/1:3 rwm
DeviceAllow=char-* m
DeviceAllow=block-* m

$ cat /run/systemd/transient/docker-a499a74aa7d90bf9d81956953c9c001fcf89d5cbcfd13af0a11db4facbcd0a01.scope.d/50-DevicePolicy.conf
# This is a drop-in unit file extension, created via "systemctl set-property"
# or an equivalent operation. Do not edit.
[Scope]
DevicePolicy=strict

(the folder paths were slightly different than given above.)

Unfortunately there's nothing obvious that occurs that triggers this. It's entirely possible that IT is pushing updates to the machine or running some kind of scan, but I have no visibility into that.

Can I take a look at your /etc/docker/daemon.json configuration? Is there any way to configure it like nvidia-ctk runtime configure --runtime=docker?

[jasonbeach]

@cdesiniotis so I spun up a new container and it actually had those extra devices yours shows:

$ cat /run/systemd/transient/docker-80155e39bfcf921e4ab7df9b4e835ffa1454c625bb6bf1c65793e82efed7c882.scope.d/50-DeviceAllow.conf
# This is a drop-in unit file extension, created via "systemctl set-property"
# or an equivalent operation. Do not edit.
[Scope]
DeviceAllow=
DeviceAllow=/dev/char/509:1 rwm
DeviceAllow=/dev/char/509:0 rwm
DeviceAllow=/dev/char/226:128 rwm
DeviceAllow=/dev/char/226:1 rwm
DeviceAllow=/dev/char/195:255 rwm
DeviceAllow=/dev/char/195:254 rwm
DeviceAllow=/dev/char/195:0 rwm
DeviceAllow=char-136 rwm
DeviceAllow=/dev/char/81:0 rwm
DeviceAllow=/dev/char/10:200 rwm
DeviceAllow=/dev/char/5:2 rwm
DeviceAllow=/dev/char/5:1 rwm
DeviceAllow=/dev/char/5:0 rwm
DeviceAllow=/dev/char/1:9 rwm
DeviceAllow=/dev/char/1:8 rwm
DeviceAllow=/dev/char/1:7 rwm
DeviceAllow=/dev/char/1:5 rwm
DeviceAllow=/dev/char/1:3 rwm
DeviceAllow=char-* m
DeviceAllow=block-* m

but the container that has been running a while (and has lost GPU access) shows only

$ cat /run/systemd/transient/docker-a499a74aa7d90bf9d81956953c9c001fcf89d5cbcfd13af0a11db4facbcd0a01.scope.d/50-DeviceAllow.conf
# This is a drop-in unit file extension, created via "systemctl set-property"
# or an equivalent operation. Do not edit.
[Scope]
DeviceAllow=
DeviceAllow=/dev/char/195:0 rwm
DeviceAllow=char-136 rwm
DeviceAllow=/dev/char/81:0 rwm
DeviceAllow=/dev/char/10:200 rwm
DeviceAllow=/dev/char/5:2 rwm
DeviceAllow=/dev/char/5:1 rwm
DeviceAllow=/dev/char/5:0 rwm
DeviceAllow=/dev/char/1:9 rwm
DeviceAllow=/dev/char/1:8 rwm
DeviceAllow=/dev/char/1:7 rwm
DeviceAllow=/dev/char/1:5 rwm
DeviceAllow=/dev/char/1:3 rwm
DeviceAllow=char-* m
DeviceAllow=block-* m

so it appears that somehow or something is modifying / overwriting that file? Out of curiosity, I restarted docker (sudo systemctl restart docker) and the long running container (a499a above) regained GPU access, but the contents of its 50-DeviceAllow file remained unchanged. So I did a sudo systemctl daemon-reload and that did trigger the GPU loss, similar to your example for that container.

The new container I spun up was unaffected by the daemon-reload. I'll leave this new container running for a while to see if it loses GPU access and if it's contents change as well.

As far as I can tell the driver I have is the proprietary nvidia driver and was installed from apt sources:

$ apt list --installed | grep nvidia

WARNING: apt does not have a stable CLI interface. Use with caution in scripts.

libnvidia-cfg1-580/unknown,now 580.126.20-1ubuntu1 amd64 [installed,automatic]
libnvidia-common-580/unknown,now 580.126.20-1ubuntu1 all [installed,automatic]
libnvidia-compute-580/unknown,now 580.126.20-1ubuntu1 amd64 [installed,automatic]
libnvidia-compute-580/unknown,now 580.126.20-1ubuntu1 i386 [installed,automatic]
libnvidia-container-tools/unknown,now 1.18.1-1 amd64 [installed,upgradable to: 1.19.0-1]
libnvidia-container1/unknown,now 1.18.1-1 amd64 [installed,upgradable to: 1.19.0-1]
libnvidia-decode-580/unknown,now 580.126.20-1ubuntu1 amd64 [installed,automatic]
libnvidia-decode-580/unknown,now 580.126.20-1ubuntu1 i386 [installed,automatic]
libnvidia-egl-gbm1/unknown,now 1.1.3-1ubuntu1 amd64 [installed,automatic]
libnvidia-egl-gbm1/unknown,now 1.1.3-1ubuntu1 i386 [installed,automatic]
libnvidia-egl-wayland1/unknown,now 1:1.1.20-1ubuntu1 amd64 [installed,upgradable to: 1:1.1.21-1ubuntu1]
libnvidia-egl-wayland1/unknown,now 1:1.1.20-1ubuntu1 i386 [installed,upgradable to: 1:1.1.21-1ubuntu1]
libnvidia-egl-xcb1/unknown,now 1.0.3-1ubuntu1 amd64 [installed,upgradable to: 1.0.5-1ubuntu1]
libnvidia-egl-xcb1/unknown,now 1.0.3-1ubuntu1 i386 [installed,upgradable to: 1.0.5-1ubuntu1]
libnvidia-egl-xlib1/unknown,now 1.0.3-1ubuntu1 amd64 [installed,upgradable to: 1.0.5-1ubuntu1]
libnvidia-egl-xlib1/unknown,now 1.0.3-1ubuntu1 i386 [installed,upgradable to: 1.0.5-1ubuntu1]
libnvidia-encode-580/unknown,now 580.126.20-1ubuntu1 amd64 [installed,automatic]
libnvidia-encode-580/unknown,now 580.126.20-1ubuntu1 i386 [installed,automatic]
libnvidia-extra-580/unknown,now 580.126.20-1ubuntu1 amd64 [installed,auto-removable]
libnvidia-fbc1-580/unknown,now 580.126.20-1ubuntu1 amd64 [installed,automatic]
libnvidia-fbc1-580/unknown,now 580.126.20-1ubuntu1 i386 [installed,automatic]
libnvidia-gl-580/unknown,now 580.126.20-1ubuntu1 amd64 [installed,automatic]
libnvidia-gl-580/unknown,now 580.126.20-1ubuntu1 i386 [installed,automatic]
libnvidia-gpucomp-580/unknown,now 580.126.20-1ubuntu1 amd64 [installed,automatic]
libnvidia-gpucomp-580/unknown,now 580.126.20-1ubuntu1 i386 [installed,automatic]
nvidia-container-toolkit-base/unknown,now 1.18.1-1 amd64 [installed,upgradable to: 1.19.0-1]
nvidia-container-toolkit/unknown,now 1.18.1-1 amd64 [installed,upgradable to: 1.19.0-1]
nvidia-dkms-580/unknown,now 580.126.20-1ubuntu1 amd64 [installed,automatic]
nvidia-driver-580/unknown,now 580.126.20-1ubuntu1 amd64 [installed,automatic]
nvidia-firmware-580-580.126.09/noble-updates,noble-security,now 580.126.09-0ubuntu0.24.04.2 amd64 [installed]
nvidia-firmware-580/unknown,now 580.126.20-1ubuntu1 amd64 [installed,automatic]
nvidia-kernel-common-580/unknown,now 580.126.20-1ubuntu1 amd64 [installed,automatic]
nvidia-kernel-source-580/unknown,now 580.126.20-1ubuntu1 amd64 [installed,automatic]
nvidia-modprobe/unknown,now 595.45.04-1ubuntu1 amd64 [installed,upgradable to: 595.58.03-1ubuntu1]
nvidia-persistenced/unknown,now 595.45.04-1ubuntu1 amd64 [installed,upgradable to: 595.58.03-1ubuntu1]
nvidia-settings/unknown,now 580.105.08-0ubuntu1 amd64 [installed,upgradable to: 595.58.03-1ubuntu1]
xserver-xorg-video-nvidia-580/unknown,now 580.126.20-1ubuntu1 amd64 [installed,automatic]

@ToviHe my /etc/docker/daemon.json is:

$ cat /etc/docker/daemon.json
{
    "dns": [
        "x.x.x.x", # redacted here
        "x.x.x.x",
        "8.8.4.4",
        "8.8.8.8"
    ],
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    }
}

I ran nvidia-ctk runtime configure --runtime=docker but it didn't change the contents of /etc/docker/daemon.json

[cdesiniotis]

So I did a sudo systemctl daemon-reload and that did trigger the GPU loss, similar to your example for that container.

The new container I spun up was unaffected by the daemon-reload.

This seems to suggest you are hitting #1133, contrary to what we initially thought. It is unclear why the long-running container has a different device allow list. I am assuming both containers were launched in identical ways (i.e. same image, same docker run command, etc.)?

[jasonbeach]

So somewhat surprisingly the new container still has access to the GPU after running a while. I looked back through my git history of the script I use to start the container and it looks like the only change I made was swapping out --gpus=all for --runtime=nvidia. So I assume I started the long running one (about 3 months now) with --gpus=all and the new one with --runtime=nvidia.

So as an experiment, I changed my script back to use --gpus=all instead of --runtime=nvidia and spun up a new container with it and it has the same shortened list of devices as my original long running container:

$ cat /run/systemd/transient/docker-eb997018a128e20abff589ea235d114a21f6556ad9e86a6ed560e0dfebd5785a.scope.d/50-DeviceAllow.conf
# This is a drop-in unit file extension, created via "systemctl set-property"
# or an equivalent operation. Do not edit.
[Scope]
DeviceAllow=
DeviceAllow=/dev/char/195:0 rwm
DeviceAllow=char-136 rwm
DeviceAllow=/dev/char/81:0 rwm
DeviceAllow=/dev/char/10:200 rwm
DeviceAllow=/dev/char/5:2 rwm
DeviceAllow=/dev/char/5:1 rwm
DeviceAllow=/dev/char/5:0 rwm
DeviceAllow=/dev/char/1:9 rwm
DeviceAllow=/dev/char/1:8 rwm
DeviceAllow=/dev/char/1:7 rwm
DeviceAllow=/dev/char/1:5 rwm
DeviceAllow=/dev/char/1:3 rwm
DeviceAllow=char-* m
DeviceAllow=block-* m

and sudo systemctl daemon-reload indeed triggers the bug in that container. I don't recall now, but I'm guessing that when I initially reported the bug, I checked to see if sudo systemctl daemon-reload triggered the behavior by spinning up a new container (which would've used --runtime=nvidia) and thus didn't exhibit the behavior.