lowvram: How to run large AI models in low VRAM GPUs

2026-11-05

Sometimes the cutting-edge AI tools require more GPU memory than consumer hardware can provide. When standard optimization fails, custom solutions are needed. Here is how I tackled the memory wall using a simple intercept layer.

The VRAM Problem: YuE2 Scenario

I recently wanted to explore YuE2: frontier music generation. The requirements were clear: Linux, Python 3.12, an NVIDIA GPU supporting BF16, and critically, 24 GB of VRAM. My setup, featuring an NVIDIA GeForce RTX 3060 with only 6 GB of VRAM, was fundamentally incompatible right out of the box.

My first attempt was to adapt the existing code, leveraging PyTorch's CPU offloading capabilities. This required deep diving into the YuE2 source code, which was time-consuming and fragile—any update to the tool broke my customized solution. I needed a solution that was agnostic, something that could run without touching the underlying application code.

The Solution: Creating lowvram

The goal was to intercept the GPU memory requests at the driver level. This led to the creation of lowvram, a tool designed to wrap CUDA programs and allow them to run even when VRAM is insufficient.

In short, lowvram intercepts CUDA calls in the library layer, trickng the application into believing that system RAM is available for use by the GPU kernel.

You can find the source code and documentation for the tool here: lowvram on GitHub.

How lowvram works

Memory Interception: It replaces the standard cuMemAlloc calls (which usually only use VRAM) with cuMemAllocManaged.
Managed Memory: This tells the NVIDIA driver to keep the data pages in VRAM as long as they are active, but to move idle pages into system RAM (Host Memory).
Transparent Usage: The application and its kernels still see standard device pointers. The driver handles the complex movement of data across the PCIe bus when necessary.
Execution: By using LD_PRELOAD or the wrapper, the tool ensures its hooks are active before the target application begins execution.

Practical Application: YuE2 and Beyond

The effectiveness of this approach was demonstrated across several use cases where memory constraints usually halt execution.

Case 1: YuE2 (Music Generation)

While the specific setup for YuE2 is complex, the philosophy holds: by using lowvram, the model attempts to load and run on a 6GB card, trading raw speed for the ability to execute the workload that would otherwise fail with a "CUDA Out of Memory" error.

Unmodified PyTorch cannot load YuE2 on this GPU. Under lowvram the same code generates 43.1 tokens per second (range over runs 41.0–46.0): 95% of the 45.3 tokens per second of the oracle, which runs the AR half from ordinary device memory after we deleted the NAR half by hand. Accelerate, the standard way to run an oversized Hugging Face model, reaches 1.9 tokens per second-lowvram is 23× faster-and the CPU 6.8.

Case 2: Llama-TTS and Qwen3-TTS

A clear, practical demonstration involved using Llama-TTS with the Qwen3-TTS model. Standard execution resulted in an immediate cudaMalloc failed: out of memory error.

By simply prepending the tool:

$ lowvram llama-tts -m Qwen3-TTS-12Hz-1.7B-Base-Q8_0.gguf -mm mmproj-Qwen3-TTS-12Hz-1.7B-Base-Q8_0.gguf --tts-lang en -p "Hello, world"

The program ran successfully.

Limitations and Considerations

While lowvram provides a powerful workaround, it is not a silver bullet. It involves fundamental changes to how the GPU interacts with system resources, and these changes come with performance trade-offs. It is crucial to be aware of these limitations:

Performance Hit: Once the working set of the model exceeds the physical VRAM capacity, data must be moved across the PCIe bus to and from system RAM. This paging mechanism significantly slows down computation.
System RAM Usage: Because it utilizes managed memory, the tool can potentially exhaust system RAM if the model attempts to allocate indefinitely, as the reported "free" memory does not shrink as managed memory is used.
Complexity: The tool is Linux-specific and requires a solid understanding of the CUDA runtime environment to utilize correctly.

Conclusion

In the realm of local AI deployment, the barrier between a perfect setup and a functional setup can often be just one line of code or one tool. If a powerful CUDA application simply won't fit your GPU, a highly practical and clever solution like lowvram offers a pathway to run it. It prioritizes functionality over peak performance, proving that sometimes, the most practical approach is the simplest one.