Working with a diverse set of open-source deep learning projects brings a unique challenge: managing the intricate web of PyTorch and CUDA dependencies. At Palaimon, we often find ourselves juggling multiple projects, each requiring a specific environment to run efficiently. This is especially critical for our autoVI visual inspection product, which relies on specific model versions. This frequently leads to what developers call “CUDA Dependency Hell.”
The Challenge: The CUDA Dependency Matrix
The core of the problem lies in the tight coupling of the deep learning stack. It’s not just about choosing a PyTorch version; it’s about ensuring every layer of the stack is compatible:
- PyTorch Versions: These are tied to specific ranges of CUDA Runtime versions.
- CUDA Runtime: These, in turn, are tied to specific NVIDIA Driver versions.
- NVIDIA Drivers: These are constrained by the physical NVIDIA Hardware (GPU) available in the system.
When you try to run a project from 2022 alongside a cutting-edge 2025 model, you quickly realize that a single host driver cannot satisfy both requirements if they fall outside the supported compatibility range.
Why Traditional Isolation Fails
Most developers reach for two common tools to solve dependency issues: Python environments (Conda/Mamba) and Docker. While excellent for many tasks, they fall short here.
Python Environments
Tools like Conda or Mamba are great for managing Python libraries and even the CUDA Runtime. However, they cannot manage the NVIDIA kernel driver. If your project requires a newer driver than what is installed on the host system, a Python environment won’t help.
Docker Isolation
Docker is often touted as the ultimate isolation tool. However, when it comes to GPUs, Docker containers share the host’s kernel and, crucially, the host’s NVIDIA driver. The nvidia-container-toolkit allows the container to access the GPU, but it still relies on the host’s driver being compatible with the CUDA version inside the container. This “leak” in isolation means you are still bound by the host’s configuration.
The Solution: VM Isolation and Dynamic GPU Passthrough
To achieve true isolation, we must move up the abstraction ladder to Virtual Machines (VMs).
By using a powerful open-source VM hypervisor like QEMU/KVM, we can create entirely independent operating systems for each project. Each VM can have its own kernel and, most importantly, its own NVIDIA driver.
Dynamic GPU Passthrough
The key to making this work is GPU passthrough (VFIO). This allows us to “detach” a GPU from the host and “attach” it directly to a VM. Using tools like virt-manager, we can manage these assignments through a user-friendly interface or remotely via CLI.
This approach offers several advantages:
- Total Isolation: One VM can run an ancient version of CUDA on Ubuntu 18.04, while another runs the latest stack on Ubuntu 24.04, all on the same physical machine.
- Reproducibility: VM images can be snapshotted and shared, ensuring that the exact environment—including the driver—is preserved.
- Hardware Flexibility: We can dynamically reassign GPUs between VMs as project needs change.
While VM-based isolation is ideal for on-premise workstations and servers, it faces challenges in the cloud. Most cloud providers (like AWS, GCP, or Azure) run their instances as VMs already. Deploying QEMU/KVM inside these instances requires nested virtualization, which is often not supported or significantly throttled.
For these scenarios, providers like AWS offer “Bare Metal” instances. These allow you to run your own hypervisor, but they come at a significantly higher cost and are often not available for flexible hourly renting, making them less ideal for short-term experimentation.
Conclusion
At Palaimon, we’ve found that while Docker and Conda are essential parts of our workflow, they aren’t a silver bullet for CUDA dependencies. For complex projects where driver versioning is a bottleneck, moving to a VM-based architecture with QEMU/KVM provides the robust isolation needed to escape dependency hell and focus on what matters: building great AI.