Fix NVIDIA-SMI 'Couldn't Communicate with the NVIDIA Driver'

Quick answer

If the NVIDIA device is still visible in PCI output but nvidia-smi cannot communicate with the driver, verify the installed driver source version and rebuild that kernel module with DKMS. This article records that recovery path and verifies it by running nvidia-smi again.

Symptoms

Originally, Ubuntu Server had already installed the NVIDIA graphics driver, and executing nvidia-smi showed that the status was normal. After installing the CUDA driver, I executed nvidia-smi to check the status, and this prompt appeared.

1
root@localhost:~# nvidia-smi
2
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

I thought the error was caused by the system not recognizing the graphics card, so I checked the PCI information.

1
root@localhost:~# lspci | grep -i nvidia
2
0b:00.0 3D controller: NVIDIA Corporation TU104GL [Tesla T4] (rev a1)

If the graphics card device is still there, then there is a problem with the driver. In this case, use dkms to compile and install the nvidia driver.

{% note info %} Dynamic Kernel Module Support (DKMS) is a program/framework that enables generating Linux kernel modules whose sources generally reside outside the kernel source tree. The concept is to have DKMS modules automatically rebuilt when a new kernel is installed.

—— From Wikipedia

{% endnote %}

Install dkms

1
root@localhost:~# apt-get install dkms

Check the NVIDIA driver version

1
root@localhost:~# ls /usr/src | grep nvidia
2
nvidia-550.25.65

Execute dkms to compile and install the NVIDIA driver module

1
root@localhost:~# dkms install -m nvidia -v 550.25.65
2
/bin/bash: /usr/local/anaconda/lib/libtinfo.so.6: no version information available (required by /bin/bash)
3
Creating symlink /var/lib/dkms/nvidia/550.25.65/source -> /usr/src/nvidia-550.25.65
4

5
Kernel preparation unnecessary for this kernel. Skipping...
6

7
Building module:
8
cleaning build area...
9
'make' -j8 NV_EXCLUDE_BUILD_MODULES='' KERNEL_UNAME=5.15.0-131-generic modules.....................
10
cleaning build area...
11

12
nvidia.ko:
13
Running module version sanity check.
14
 - Original module
15
   - No original module exists within this kernel
16
 - Installation
17
   - Installing to /lib/modules/5.15.0-131-generic/updates/dkms/
18

19
nvidia-uvm.ko:
20
Running module version sanity check.
21
 - Original module
22
   - No original module exists within this kernel
23
 - Installation
24
   - Installing to /lib/modules/5.15.0-131-generic/updates/dkms/
25

26
nvidia-modeset.ko:
27
Running module version sanity check.
28
 - Original module
29
   - No original module exists within this kernel
30
 - Installation
31
   - Installing to /lib/modules/5.15.0-131-generic/updates/dkms/
32

33
nvidia-drm.ko:
34
Running module version sanity check.
35
 - Original module
36
   - No original module exists within this kernel
37
 - Installation
38
   - Installing to /lib/modules/5.15.0-131-generic/updates/dkms/
39

40
nvidia-peermem.ko:
41
Running module version sanity check.
42
 - Original module
43
   - No original module exists within this kernel
44
 - Installation
45
   - Installing to /lib/modules/5.15.0-131-generic/updates/dkms/
46

47
depmod....
48
root@localhost:~#

View NVIDIA driver information

1
root@localhost:~# nvidia-smi
2
Thu Feb 20 15:11:42 2025
3
+-----------------------------------------------------------------------------------------+
4
| NVIDIA-SMI 550.25.65              Driver Version: 550.25.65      CUDA Version: 12.8     |
5
|-----------------------------------------+------------------------+----------------------+
6
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
7
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
8
|                                         |                        |               MIG M. |
9
|=========================================+========================+======================|
10
|   0  Tesla T4                       Off |   00000000:0B:00.0 Off |                    0 |
11
| N/A   56C    P0             26W /   70W |       1MiB /  15360MiB |      9%      Default |
12
|                                         |                        |                  N/A |
13
+-----------------------------------------+------------------------+----------------------+
14

15
+-----------------------------------------------------------------------------------------+
16
| Processes:                                                                              |
17
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
18
|        ID   ID                                                               Usage      |
19
|=========================================================================================|
20
|  No running processes found                                                             |
21
+-----------------------------------------------------------------------------------------+

Displays normally, perfect!

For the initial driver, CUDA, and cuDNN installation path, see Install NVIDIA drivers, CUDA, and cuDNN on Linux.

Fix NVIDIA-SMI 'Couldn't Communicate with the NVIDIA Driver'

https://www.koevn.com/p/dc535881/

Author Koevn

Published February 24, 2025

License CC BY-NC-SA 4.0