Q4, Q5, Q8, FP16: which quantization should you download?

Open a model page on Hugging Face or in LM Studio and you are hit with a wall of file names: Q4_K_M, Q5_K_S, Q6_K, Q8_0, F16. They are all the same model stored at different precision. Picking the right one is the difference between a model that fits and one that doesn't.

What the numbers mean

The number after Q is roughly the bits per weight. FP16 keeps every weight as a 16-bit number and is the reference quality. Q8 halves the size with no noticeable loss. Q5 and Q6 are the sweet spot for quality per gigabyte. Q4 is the most popular home choice: a quarter of the FP16 size, and on models above 7B the quality loss is hard to notice in normal use. Q3 and Q2 exist but degrade clearly; use them only to squeeze a model that otherwise cannot run at all.

The letters matter less. K means the newer "k-quant" method, which is smarter about which weights get more bits. S, M and L are small, medium and large variants of the same bit depth. If in doubt, Q4_K_M is the default most guides recommend.

A rule of thumb

Bigger model at lower precision beats smaller model at higher precision. A 14B model at Q4 will generally outperform an 8B model at Q8, and they take about the same memory. So pick the largest model that fits at Q4 or Q5 before considering a higher quantization of something smaller.

Speed is affected too

Generation speed is limited by how fast your card can read the weights from memory. Smaller files read faster, so Q4 is also the quickest option. On a 12 GB card, an 8B model might run at 45 tokens per second at Q4 and 30 at Q8.

Our defaults

The checker defaults to "best that fits", which tries Q8 first and steps down to Q5 then Q4. If you want to force a specific precision, change the Quantization dropdown and the whole list updates.