Most people default to Q4_K_M in llama.cpp because it's the "safe" choice. But I've found the real win comes from testing your actual workflow. A 70B model in Q3_K_S cuts latency significantly compared to Q4_K_M on the same hardware, with imperceptible quality loss for most tasks. The bottleneck becomes memory bandwidth, not raw VRAM size.
Here's what changed my setup: I stopped chasing maximum quality and started measuring latency on real prompts. A 4-bit quantized Mistral answers coding questions as well as the full-precision version, but returns results faster. For summarization or creative writing, Q5 variants matter more. For RAG or classification tasks, I can drop to Q3 without noticing the difference.
The catch is context length. Lower quantization plus longer context means RAM pressure. If you're doing 4K+ context windows, you can't always drop to the most aggressive quantization. That's where the tradeoff gets real.
Spend an hour profiling your use case with different quantization levels. Measure latency, memory usage, and quality on a few real prompts. You'll find your sweet spot isn't where you started. Mine shifted twice in six months.
Top comments (1)
The memory-bandwidth point is the one that took me embarrassingly long to internalise. On a consumer GPU with a 70B you're almost never compute-bound during decode; you're moving weights across the bus every single token, and a smaller quant means fewer bytes per token which directly translates to higher tok/s. Q3_K_S vs Q4_K_M is a great example because the perplexity gap on most real tasks is well inside the noise you'd see from changing your sampler.
One thing I'd add: task sensitivity to quantization is wildly non-uniform. Coding and structured extraction tolerate aggressive quantization remarkably well. Long-form reasoning and anything that depends on rare-token recall (named entities, numbers, code identifiers in unfamiliar libs) degrades much faster, and the degradation is the sneaky kind — the output still looks fluent, it's just subtly wrong. I keep a tiny eval set per use case for exactly this reason; vibe-checking quants is how you ship something that works for you and silently fails for everyone else.
Also worth flagging: imatrix-calibrated quants (the ones that use an importance matrix from a calibration corpus) often outperform vanilla K-quants at the same bit width, especially in the Q3 range. If you haven't tried them yet, it's basically a free quality bump for the cost of one calibration run.