Home/FAQ & Glossary/Why does a 70B model crash on a 32GB graphics card?
FREQUENTLY ASKED QUESTIONconcepts
← Back to FAQ & Glossary

Why does a 70B model crash on a 32GB graphics card?

A standard 4-bit (Q4_K_M) 70B model takes up 40GB to 42GB on disk. That is already bigger than a 32GB graphics card like an RTX 5090 before you even type a single word.

Even if you drop down to an aggressively compressed 2-bit or 3-bit quant (roughly 26GB to 30GB) so the file technically fits inside 32GB, loading the model weights is only part of the equation. Your GPU still needs spare VRAM for three other things:

  • The Context Window (KV Cache): As your chat grows or you paste in long documents, the GPU holds that history in VRAM. On a 70B model, an 8,000-word conversation eats roughly 2.5GB to 3GB. Crank that up to a 128K context window, and the cache alone demands over 40GB.
  • Compute & Activation Buffers: Generating tokens requires temporary scratchpad memory for CUDA kernels and framework overhead, adding another 1GB to 2GB.
  • Your Monitor & Operating System: Windows or Linux reserves 0.5GB to 1.5GB of VRAM just to render your desktop, browser tabs, and open apps.

The instant total demand crosses 32.0GB, the driver runs out of space and throws a CUDA Out of Memory (OOM) crash.

How to run a 70B model on a 32GB GPU without crashing

  1. Use partial CPU/RAM offloading: In tools like LM Studio or Ollama, put roughly 50 to 60 of the model's 80 layers onto your 32GB GPU and spill the rest into system RAM (assuming you have 64GB+ of system RAM). Speeds will drop to around 4–8 words per second instead of 25+, but the session stays stable.
  2. Cap your context length: Lock your context window to 4,096 or 8,192 tokens rather than the 128,000 maximum so the KV cache cannot balloon out of control.
  3. Add a second GPU: Keeping a 70B Q4_K_M model 100% in fast VRAM takes 48GB total. That is why local AI builders pair two 24GB cards (such as dual RTX 3090s or 4090s) or add a 16GB–24GB card alongside a 32GB GPU.

As a general rule, leave at least 15% to 20% of your VRAM empty when choosing a model file so your chat history and desktop have room to breathe.

Looking for verified hardware rigs?

Check our real-time £/GB VRAM index to find systems optimized for local model inference.

Explore VRAM Index →
Rent Cloud GPU Power

Not Ready to Buy a Full Rig?

Don't want to spend £3,500+ upfront? Rent the GPU power instead.

Buying a dedicated AI workstation is a serious investment—and if you aren't ready to spend that much right now, you don't have to. Renting cloud GPUs gives you instant access to high-VRAM hardware like RTX 4090s and H100s on demand. Spin up an instance for fine-tuning or running massive 70B+ models, pay strictly for the minutes you use, and avoid the upfront hardware cost, electricity bills, and maintenance.

Vast.ai Global GPU Marketplace

Lowest $/hr
Unbeatable $/hr rates on community-hosted RTX 4090 & 3090 rigs
Deploy Pod →

RunPod Community & Secure Cloud

Community & Secure Cloud
RTX 4090 from $0.34/hr • H100 SXM from $1.99/hr (Billed in USD)
Deploy Pod →

FindBestRig independently benchmarks cloud GPU providers to compare rental economics against physical hardware.