AI & AgentsEngineering note
"Driver/library version mismatch": How an Automatic Update Broke Our GPU Server Three Times
Automatic updates install the new NVIDIA user-space files at once, but the old kernel module stays loaded until a restart. Everything already running keeps working, which is why the problem shows up days later, at the worst moment.
Author
CrpyByte Engineering
Published
Reading time
3 min read
We run an AI chatbot on a cloud server with one NVIDIA T4 GPU. The GPU hosts the embedding model, which turns every question into numbers before the search. On 29 May 2026 every chat question started hanging for about three minutes and then returned nothing. The embedding service had crashed, and its replacement could not start. The error in its log:
Failed to initialize NVML: Driver/library version mismatchNobody had touched the GPU driver. An automatic update had, eight days earlier.
What actually happened
- 21 May, 06:00. Ubuntu's unattended upgrades installed a newer NVIDIA driver package, 580.126.09 to 580.159.03. That replaces the user-space libraries on disk. The kernel module stays the old version until the machine restarts or the module is reloaded. The server had been up for 48 days and was not restarted.
- 21 to 28 May. Nothing visibly wrong. The embedding service had loaded the GPU long before the update and kept using it.
- 29 May. A memory setting we had changed on the database left the server short of memory, the embedding service was stopped by the system, and the new copy tried to open the GPU from scratch. New libraries met the old kernel module, and the driver's management library (NVML) refused to start. Chat was down.
This is why the error is so confusing: the update and the failure are days apart, and the failure appears to be caused by whatever restarted last.
How to confirm it in 30 seconds
Compare the version of the module that is loaded with the version installed on disk:
cat /proc/driver/nvidia/version # kernel module currently loaded
modinfo nvidia | grep ^version # module installed on disk
nvidia-smi # fails with the mismatch errorIf the two versions differ, that is the whole problem. Nothing is broken; the two halves of the driver are out of step.
The fix without a reboot
A restart fixes it, but a restart of a busy server is not always safe. Ours had a second kernel installed that would have become the default on reboot and had never been tested with the GPU. Reloading the NVIDIA kernel module does the same job:
sudo systemctl stop <every service using the GPU>
sudo fuser -v /dev/nvidia* # must show nothing
sudo modprobe -r nvidia_uvm nvidia_modeset nvidia
sudo modprobe nvidia nvidia_uvm
nvidia-smi # works again, new version
sudo systemctl start <your GPU services>Unloading only works when nothing holds the GPU, which is what the fuser check is for. On 29 May we first had to free memory so the stuck embedding workers could exit; after that the reload took seconds. The embedding model loaded again at about 0.04 seconds per question, and chat came back at 10:35 UTC, about 40 minutes after the crash.
It came back twice more
We fixed the symptom and not the cause, so the same update path broke it again on 25 July. That time the embedding model kept working: the CUDA library tolerated the small version gap and only NVML, which nvidia-smi and monitoring tools use, insisted on an exact match. Monitoring went blind while the service ran. Three occurrences were enough.
On 29 July we added two guards so the driver only changes when we decide:
# 1. Hold the installed NVIDIA packages at their current version
sudo apt-mark hold $(dpkg -l | awk '/^ii/ && /nvidia/ {print $2}')
# 2. Tell unattended-upgrades never to touch them
# /etc/apt/apt.conf.d/51-nvidia-hold-blacklist.conf
Unattended-Upgrade::Package-Blacklist {
"nvidia-";
"libnvidia-";
};Security updates for everything else still install automatically. Driver updates now happen in a planned window, with a module reload or a restart straight after, while someone is watching.
Checklist for a GPU server
- Hold the NVIDIA packages and blacklist them in unattended-upgrades. Update them on purpose.
- After any driver update, reload the module or restart the same day, not whenever the server next happens to reboot.
- Compare the loaded and installed driver versions (
/proc/driver/nvidia/versionandmodinfo nvidia) in your health check, and alert when they differ. - Know which kernel boots next before you rely on a restart.
- Leave memory headroom for services that load models at start. A restart under memory pressure is often what triggers the failure.
- Write down the reload steps where the person on call can find them.
If you run your own models on a GPU server and want it checked before it fails at a bad moment, tell us about the setup.