| id | TD-007 | ||
|---|---|---|---|
| title | Q4_K_M model quantization by default, Q5_K_M/Q6 for coding | ||
| status | accepted | ||
| date | 2026-06-09 | ||
| supersedes | |||
| superseded_by | |||
| tags |
|
Weight quantization trades precision for footprint/speed. The general rule: parameters are the expensive axis of quality, precision the cheap one — a 14B-Q4 is smarter than an 8B-Q8. But the degradation from quantization is not uniform across tasks.
Default Q4_K_M for the MoE and for general chat. For coding and tight instruction-following, step up to Q5_K_M/Q6.
Q4_K_M takes <2% off a model on knowledge tasks and keeps the footprint within the MoE's ~18 GB. But coding and precise instruction-following degrade faster under Q4: if the use case is programming, Q5_K_M/Q6 recovers that margin at the cost of a few more GB.
- Positive: a lightweight default that fits in RAM; a clear path for those who code (step up the quant).
- Negative: Q5/Q6 increase the footprint and approach/exceed the RAM ceiling on the MoE → to be assessed case by case.
Rejected: too high a footprint for the MoE (it does not fit) and a marginal quality gain on common tasks.
Q4_0 is needed for the NPU/llama.cpp-OpenVINO path (TD-012), but for the iGPU the k-quants (Q4_K_M+) are better at the same bit count.