Quantization Error: How Low Can the Bits Go Before Quality Breaks

A large language model is, to first approximation, a pile of weight matrices, and serving it quickly is mostly a problem of moving those matrices out of memory fast enough. Store each weight in sixteen bits and every forward pass must stream sixteen bits per parameter; store it in four and you move a quarter as much. That is the entire appeal of quantization, and it is a very good deal, right up until the rounding you introduced changes what the model says. The difference between a four-bit model that is indistinguishable from the original and one that produces garbage is not mysterious. It is governed by how large the rounding error is, where you place the quantization grid, and which error you chose to minimize. This is the math that decides how low the bits can go.

What quantization actually is

Define the operation precisely before reasoning about its error. Uniform quantization maps a real-valued weight w onto one of a small number of evenly spaced levels. Pick a scale s (the spacing between levels) and a zero-point z (the integer level that represents zero), then store the integer q obtained by dividing by the scale, rounding, and clipping into range. To use the weight you dequantize: multiply the stored integer back by the scale. The stored value is an integer in a small range; the reconstructed value is an approximation of the original real number.

With b bits you have 2^b levels. If the weights you are quantizing span the range from w_min to w_max, spreading 2^b levels evenly across that range fixes the scale as the width of the range divided by the number of gaps, 2^b minus one. Everything about the error follows from that single number s: it is the width of one grid cell, and no reconstructed weight can be more than half a cell away from the truth.

This is 'post-training' quantization: the model was trained in high precision and is compressed afterward, without gradient updates. The alternative, training in low precision, is a different discipline; here the weights are fixed and the only freedom is how cleverly you round them.

The size of a rounding error

Rounding to the nearest grid point can miss the true value by at most half a cell, so the absolute quantization error is bounded by s over two. That is the worst case. To reason about typical behaviour, model the error as a random variable. Under the standard high-resolution assumption, when the grid is fine relative to how fast the weight distribution changes, the rounding error is well approximated as uniform on the interval from minus s over two to plus s over two.

A uniform random variable on an interval of width s has variance equal to s squared over twelve. This is the quantization noise power, and it is the quantity that matters, because the damage a quantized layer does to the model's output scales with the variance of the per-weight error, not its worst case. Halve the step s and you quarter the noise variance.

Two facts are already visible. First, error scales with the range you must cover: a layer whose weights span a wide interval gets a large step and therefore large error at the same bit-width. Second, error falls quadratically as the step shrinks, which is what makes each additional bit so valuable.

Six decibels per bit

Adding one bit doubles the number of levels, which halves the step s, which quarters the noise variance. Expressed as a ratio of signal power to quantization-noise power, and converted to decibels, quartering the noise raises the signal-to-quantization-noise ratio by about six decibels. The classic result for a full-scale uniform quantizer is that the ratio is approximately six-point-oh-two times the number of bits, plus one-point-seven-six decibels.

This is the single most useful number in the subject. It says precision buys quality on a fixed exchange rate: every bit you remove costs roughly six decibels of fidelity. Going from sixteen-bit to eight-bit surrenders about forty-eight decibels and is usually invisible because you had an enormous surplus. Going from eight-bit to four-bit surrenders another twenty-four, and that is where the surplus runs out for many layers.

The rule is an idealization: it assumes the weights fill the range roughly uniformly and that the noise model holds. Real weight distributions are peaked and have tails, so the realized fidelity is lower than the formula promises, especially at low bit-width, but the six-decibels-per-bit slope is a reliable guide to the trade.

Granularity: where you put the scale

The step s is set by the range of the weights sharing a scale, so the cheapest way to cut error is to let fewer weights share a scale. Per-tensor quantization uses one scale for an entire weight matrix, so a single wide-ranging column forces a coarse grid on everything. Per-channel quantization gives each output channel its own scale, so a well-behaved channel is not punished for a noisy neighbour. Group-wise quantization goes finer still, giving every contiguous group of, say, 64 or 128 weights its own scale.

Finer granularity shrinks the range each scale must cover, which shrinks s, which quadratically shrinks the noise. The cost is a small amount of extra metadata: one scale (and possibly one zero-point) per group instead of per tensor. In practice group-wise quantization is what makes four-bit weights viable at all, because it keeps the per-group range small enough that the six-decibels-per-bit budget still buys acceptable fidelity.

There is a real trade here, not a free lunch: shrink the group too far and the scale metadata itself starts to consume the memory you were trying to save, and the effective bit-width creeps back up.

Outliers wreck the range

The formulas assume the range is set by the bulk of the weights, but transformer weight and activation distributions have heavy tails: a small number of coordinates carry values far larger than the rest. Because the step s is proportional to the full range, a handful of outliers drags the grid coarse for the overwhelming majority of ordinary weights, inflating everyone's error to protect a few extreme values.

Activation-aware weight quantization, AWQ (Lin et al., 2023), makes a sharp observation: not all weights matter equally to the output, and the ones that matter can be identified by the magnitude of the activations they multiply. AWQ scales up those salient weight channels before quantizing and compensates on the activation side, so the important channels land on a finer effective grid while the range for the rest stays tight. The saliency signal is the activation statistics, not the weights alone, which is why the method is called activation-aware.

The general principle this illustrates is that a uniform grid is the wrong tool for a non-uniform distribution, and the fix is to reshape the problem, by rescaling or grouping, so that within each scale the values you quantize are as uniform and tightly-ranged as you can make them.

Minimize the right error

Rounding each weight to its nearest grid point minimizes the error in the weights. But the model does not care about the weights; it cares about the layer's output. Two roundings with the same total weight error can do very different damage to the product of the weight matrix with its input, because inputs are not uniform and errors in high-leverage directions matter more.

GPTQ (Frantar et al., 2022) acts on exactly this distinction. Instead of minimizing the change in the weights, it minimizes the change in the layer's output, the squared error between the original weight matrix times its inputs and the quantized weight matrix times the same inputs. It quantizes one column at a time and, crucially, propagates the rounding error just introduced into the not-yet-quantized columns so they can compensate, using second-order information (the input correlation matrix, a Hessian of this least-squares objective) to decide how. The result is a rounding that is deliberately not nearest-neighbour where nearest-neighbour would hurt the output.

The lesson generalizes beyond any one method: the objective you optimize should be the quantity the system is sensitive to. Minimizing weight error is a proxy; minimizing output error is the target, and closing that gap is where most of the modern gains at four bits come from.

When it breaks

The failure modes are the mirror image of the levers. First, too few bits: the noise variance grows as the step squared, so as you drop below about four bits the per-weight error stops being a small perturbation and becomes comparable to the signal. The six-decibels-per-bit budget simply runs out, and no rounding cleverness recovers a fidelity the bits cannot represent.

Second, untamed outliers: if a per-tensor scale is set by a few extreme coordinates, the effective bit-width for the bulk of the weights collapses even though the nominal bit-width is unchanged. This is why per-tensor four-bit quantization often fails where group-wise or activation-aware four-bit succeeds on the same model.

Third, error accumulation: a model is a stack of layers, and each quantized layer perturbs the input to the next. Even if the per-layer output error is small, small errors compound through depth, so a bit-width that is safe for one layer in isolation can drift the final output more than expected. The defensive posture is to measure the end-to-end output error, not just the per-weight or even per-layer error, and to spend precision unevenly, keeping sensitive layers higher and compressing robust ones harder.

Practitioner Toolkit

Copy-paste, strictly defensive artifacts you can use today. Nothing here attacks a real system.

for layer in model.linear_layers:
    X = calibration_activations[layer]        # representative inputs
    W = layer.weight
    W_hat = quantize(W, bits=b, group=g)
    num = frobenius(W @ X - W_hat @ X)
    den = frobenius(W @ X) + 1e-9
    report(layer.name, 'rel_output_error', num / den)
# raise bit-width on the worst layers, lower on the best

Glossary

Uniform quantization

Mapping real values onto evenly spaced levels defined by a scale and a zero-point.

Scale (s)

The spacing between adjacent quantization levels; equals the value range divided by the number of gaps.

Zero-point (z)

The integer level that represents the real value zero, allowing asymmetric ranges.

Quantization noise

The rounding error treated as a random variable; for a uniform quantizer its variance is s squared over twelve.

SQNR

Signal-to-quantization-noise ratio, the power of the signal relative to the rounding noise, rising about six decibels per bit.

Group-wise quantization

Assigning a separate scale to each small contiguous group of weights to shrink the per-group range and error.

Activation-aware quantization

Selecting and protecting the weight channels that matter most to the output, identified from activation magnitudes.

References

1. Frantar et al. (2022), GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
2. Lin et al. (2023), AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration
3. Vaswani et al. (2017), Attention Is All You Need (NeurIPS)
4. Kwon et al. (2023), Efficient Memory Management for LLM Serving with PagedAttention (SOSP)