KV Cache (Key-Value Cache)
The KV Cache (Key-Value Cache) is the extra slice of VRAM your system reserves to remember your ongoing conversation so the AI does not have to re-read the entire chat from scratch after every single word it writes.
When you first load a model, the weights take up a fixed chunk of memory. As soon as you start chatting, pasting logs, or uploading documents, the KV Cache grows dynamically alongside your context window. The longer the thread gets, the larger that cache becomes.
This is the hidden trap that catches many beginners off guard: a model file might be 14GB and look like it fits comfortably inside a 16GB GPU, only to spill over into slow system RAM—or crash with an Out-Of-Memory error—once your conversation hits 10,000 tokens. Keeping your context length sensible or enabling KV cache quantization in tools like Ollama or LM Studio keeps that memory footprint under control.
Looking for verified hardware rigs?
Check our real-time £/GB VRAM index to find systems optimized for local model inference.