Quantization for hardware

In short: Quantization shrinks neural-network weights and activations from 32-bit floats to compact integers (int8) or half-precision floats (fp16) so hardware runs them faster and with far less power — post-training quantization is the quick, calibration-only path, while quantization-aware training recovers the last fraction of accuracy that aggressive int8 would otherwise lose.

Why hardware cares about number formats

A trained network is, at heart, a huge pile of multiply-accumulate (MAC) operations. The cost of each MAC — in silicon area, energy, and latency — is dominated by the width of the numbers it chews on. An fp32 multiplier is a large, power-hungry block; an int8 multiplier is roughly an order of magnitude cheaper. Moving a 32-bit word in and out of DRAM also costs far more energy than the arithmetic itself, so halving or quartering the data width is a double win: smaller compute units and less memory traffic.

Rule of thumb from silicon studies: an int8 MAC consumes on the order of 0.2 pJ versus roughly 3.7 pJ for an fp32 MAC in a mature process node, and a 32-bit DRAM access can cost hundreds of pJ. This is why every modern neural processing unit (NPU) is built around int8 (and increasingly int4) datapaths.

The question quantization answers is: how do we represent a weight that was trained as 0.4213... in fp32 using an 8-bit integer without wrecking accuracy? The trick is a uniform affine mapping.

The affine mapping: scale and zero-point

Int8 quantization maps a real value r to an integer q using two parameters per tensor (or per channel): a floating-point scale S and an integer zero-point Z:

• q = round(r / S) + Z  (quantize)
• r ≈ S · (q − Z)  (dequantize)

For symmetric int8 weights, Z = 0 and the range is [−127, 127]; S is simply max(|w|)/127. Activations are often asymmetric (a ReLU output is never negative), so the zero-point lets the integer range land exactly on the data's true min/max, wasting no codes.

int8 vs fp16: two different tradeoffs

These are not competitors so much as tools for different jobs.

fp16 keeps a floating exponent, so it preserves dynamic range and needs no calibration — you essentially cast and go. Its weakness is that the 5-bit exponent can overflow or underflow on large accumulations, which is why mixed-precision training keeps a master fp32 copy of weights and a loss-scaling factor. int8 throws away the exponent entirely in exchange for a 4× memory cut and much cheaper MACs, but a single fixed scale per tensor means outlier values can swamp the resolution available to everything else.

Post-training quantization (PTQ)

PTQ takes an already-trained fp32 model and quantizes it with no gradient updates. The only work is calibration: run a few hundred representative inputs through the network, record the min/max (or a percentile/KL-divergence clip) of each activation tensor, and compute the scales. Vendor toolchains make this a one-command step — trtexec / TensorRT's calibrator, ONNX Runtime's quantization API, or the quantization flows inside Synopsys and Cadence AI-accelerator design kits that map the graph onto a fixed-point datapath.

PTQ on a well-behaved CNN like ResNet-50 typically loses under 1% top-1 accuracy going to int8. The same recipe on a transformer can drop several points because of large activation outliers in the attention projections — the classic sign you need per-channel scales or SmoothQuant-style equalization.

PTQ variants worth knowing: per-tensor (one scale for the whole weight matrix, cheapest), per-channel (one scale per output filter, much better for convolutions), and weight-only quantization (int4 weights, fp16 activations) which is the dominant pattern for large language models where memory bandwidth, not MAC count, is the bottleneck.

Quantization-aware training (QAT)

When PTQ leaves too much accuracy on the table, QAT inserts "fake quantization" nodes into the training graph. During the forward pass, weights and activations are rounded to their int8 grid; during the backward pass, the non-differentiable rounding is bypassed with a straight-through estimator (the gradient passes through as if the rounding were identity). The network therefore learns weights that are robust to the quantization grid, often clawing back nearly all the lost accuracy.

QAT costs real training time and data, so the practical decision tree is simple: try PTQ first; reach for QAT only when the accuracy gap after calibration is unacceptable for the target application.

The accuracy-vs-efficiency tradeoff in practice

Every bit you remove buys efficiency and spends accuracy headroom. The engineering art is spending it where the network does not care:

• Keep sensitive layers wide. The first and last layers, and softmax/attention logits, are frequently left in fp16 while the bulk of the MACs run int8 — a mixed-precision graph.
• Per-channel beats per-tensor for weights at essentially no hardware cost, because the scale folds into the layer's output requantization.
• Fuse before you quantize. Folding batch-norm into the preceding convolution removes a scaling step that would otherwise double the quantization error.
• Validate on real data. A model that looks fine on calibration images can degrade on edge cases; teams that build this skill end up doing serious silicon work for semiconductor design services firms where a 1% accuracy regression can fail a customer spec.

For a deeper look at how these quantized graphs are scheduled onto real silicon, see our companion piece on dataflow accelerator architecture, and the broader course catalog at CourseTron's AI-hardware courses.

Want to go from "I can define int8" to "I can quantize a transformer and hit the latency budget"? CourseTron's AI Hardware & Edge Inference track walks you through PTQ, QAT, and silicon mapping with hands-on vendor toolchains — exactly the skills VLSI and ML-systems interviewers probe.

Frequently asked questions

Is int8 always 4× faster than fp32?

Not automatically. You get up to 4× less memory and far cheaper MACs, but real speedup depends on the hardware actually having int8 execution units (tensor cores, DSP MACs, or an NPU) and on the model being memory- or compute-bound. A tiny model dominated by overhead may see little gain.

When should I use QAT instead of PTQ?

Use PTQ first — it needs only calibration data and minutes of work. Switch to QAT when post-calibration accuracy loss exceeds your tolerance, which is common for transformers, very low-bit targets (int4), or compact networks like MobileNet where every parameter matters.

What is the difference between fp16 and bf16?

Both are 16-bit, but fp16 spends 10 bits on the mantissa and 5 on the exponent, while bf16 keeps 8 exponent bits (matching fp32's range) and only 7 mantissa bits. bf16 trades precision for dynamic range, which makes it far easier to train with without overflow; fp16 is more precise but needs loss scaling.

Does quantization help large language models?

Yes, massively — LLM inference is bandwidth-bound, so int8 or int4 weight-only quantization (GPTQ, AWQ) can cut model size 2–4× and dramatically raise tokens/second with minimal quality loss, because activations can stay in fp16 while the giant weight matrices shrink.