Home/FAQ & Glossary/What is \"Quantization\" (like Q4_K_M or Q8)?
FREQUENTLY ASKED QUESTIONconcepts
← Back to FAQ & Glossary

What is \"Quantization\" (like Q4_K_M or Q8)?

Quantization is essentially JPEG compression for AI models. Out of the box, models store their weights as high-precision 16-bit numbers (FP16). That makes them huge—an 8-billion-parameter model needs roughly 16GB of VRAM at full precision. Quantization rounds those numbers down into smaller 8-bit or 4-bit integers, shrinking the file by 50% to 75% without wrecking the model's smarts. Because the file is smaller, your GPU reads it faster and spits out text quicker.

When you download models in apps like LM Studio or Ollama, the suffix at the end of the filename tells you how aggressively it's been compressed:

LabelPrecision8B Model Size (incl. context)Best For
Q4_K_M (Recommended)4-bit (Mixed)~5.5 – 6 GBThe sweet spot. Cuts size by ~70% with less than 1–3% quality loss, fitting comfortably on 8GB GPUs.
Q5_K_M / Q6_K5-bit / 6-bit~6.5 – 7.5 GBA solid middle ground if you're running a 12GB–16GB GPU with VRAM to spare.
Q8_08-bit~9 – 10 GBVirtually indistinguishable from the original unquantized model; worth grabbing if you have plenty of headroom.

What does _K_M actually mean? In modern GGUF model files, K means the quantization method protects the model's most critical "thinking" layers at higher precision while squeezing less sensitive layers harder. The M stands for Medium.

If you have to choose, always run a larger model at Q4_K_M over a smaller model at Q8. A 14B model at Q4_K_M will noticeably outperform an 8B model at Q8 while eating up roughly the same amount of VRAM.

Looking for verified hardware rigs?

Check our real-time £/GB VRAM index to find systems optimized for local model inference.

Explore VRAM Index →
Rent Cloud GPU Power

Not Ready to Buy a Full Rig?

Don't want to spend £3,500+ upfront? Rent the GPU power instead.

Buying a dedicated AI workstation is a serious investment—and if you aren't ready to spend that much right now, you don't have to. Renting cloud GPUs gives you instant access to high-VRAM hardware like RTX 4090s and H100s on demand. Spin up an instance for fine-tuning or running massive 70B+ models, pay strictly for the minutes you use, and avoid the upfront hardware cost, electricity bills, and maintenance.

Vast.ai Global GPU Marketplace

Lowest $/hr
Unbeatable $/hr rates on community-hosted RTX 4090 & 3090 rigs
Deploy Pod →

RunPod Community & Secure Cloud

Community & Secure Cloud
RTX 4090 from $0.34/hr • H100 SXM from $1.99/hr (Billed in USD)
Deploy Pod →

FindBestRig independently benchmarks cloud GPU providers to compare rental economics against physical hardware.