Home/FAQ & Glossary/Quantization (Q4, Q4_K_M, Q8, Unquantized/FP16)
GLOSSARY DEFINITIONConcepts
← Back to FAQ & Glossary

Quantization (Q4, Q4_K_M, Q8, Unquantized/FP16)

Quantization is the process of shrinking an AI model so it fits into consumer graphics cards without losing much of its intelligence. It works by rounding off the precision of the numbers (weights) inside the model—much like saving a massive raw photo as a high-quality JPEG.

Models are normally trained in FP16 (Unquantized / 16-bit) format, where each parameter takes up 2 bytes of memory. That translates to roughly 2GB of VRAM per billion parameters. Under FP16, a 70B model demands over 140GB of VRAM, putting it well out of reach for normal desktop hardware.

By compressing those numbers down to 4-bit quantization (Q4)—with Q4_K_M being the most popular balanced preset—the footprint drops to roughly 0.6GB per billion parameters. Suddenly, that same 70B model squeezes cleanly onto a 48GB or 64GB Mac or dual-GPU workstation with almost no noticeable drop in answer quality. Q8 (8-bit) sits in the middle at around 1GB per billion parameters, offering near-identical accuracy to unquantized models if you have the extra memory headroom.

Looking for verified hardware rigs?

Check our real-time £/GB VRAM index to find systems optimized for local model inference.

Explore VRAM Index →
Rent Cloud GPU Power

Not Ready to Buy a Full Rig?

Don't want to spend £3,500+ upfront? Rent the GPU power instead.

Buying a dedicated AI workstation is a serious investment—and if you aren't ready to spend that much right now, you don't have to. Renting cloud GPUs gives you instant access to high-VRAM hardware like RTX 4090s and H100s on demand. Spin up an instance for fine-tuning or running massive 70B+ models, pay strictly for the minutes you use, and avoid the upfront hardware cost, electricity bills, and maintenance.

Vast.ai Global GPU Marketplace

Lowest $/hr
Unbeatable $/hr rates on community-hosted RTX 4090 & 3090 rigs
Deploy Pod →

RunPod Community & Secure Cloud

Community & Secure Cloud
RTX 4090 from $0.34/hr • H100 SXM from $1.99/hr (Billed in USD)
Deploy Pod →

FindBestRig independently benchmarks cloud GPU providers to compare rental economics against physical hardware.