In short: Mixed-precision training runs the heavy matrix math in 16-bit FP16 or BF16 to exploit a GPU's tensor cores for 2–8× speedups and roughly half the memory, while keeping a FP32 "master" copy of the weights (and loss scaling for FP16) so tiny gradient updates are never rounded away.
Why 16 bits, and why not all 16 bits
Deep-learning accelerators are bottlenecked by two things: how many multiply-accumulate operations they can push per second, and how fast they can move operands in and out of memory. A 32-bit float (FP32) is the comfortable default for numerical stability, but it is twice the data to move and does not get to use the dedicated low-precision matrix units that modern silicon dedicates most of its die area to. Mixed-precision training is the engineering compromise: do the expensive, dense linear algebra in 16-bit formats, but hold the state that must not drift in 32-bit.
The reason you cannot simply train everything in 16 bits is dynamic range and rounding. Weight updates in a late-stage transformer can be on the order of 10-7 relative to the weight itself. In FP16 that update rounds to zero because the number of mantissa bits is too small to represent "weight plus a tiny delta". Over thousands of steps the model simply stops learning. The master-weights-plus-loss-scaling recipe exists precisely to defeat that failure mode.
FP16 vs BF16: the format that changes everything
Both are 16-bit, but they split those bits differently, and that single design choice dictates your whole training recipe.
| Format | Exponent bits | Mantissa bits | Approx. range | Decimal precision |
|---|---|---|---|---|
| FP32 | 8 | 23 | ~1e±38 | ~7 digits |
| FP16 (IEEE half) | 5 | 10 | ~6e-8 to 65 504 | ~3 digits |
| BF16 (bfloat16) | 8 | 7 | ~1e±38 | ~2 digits |
BF16 keeps FP32's full 8-bit exponent, so it has the same dynamic range as FP32 and simply loses precision. FP16 trades range for precision. The practical consequence: BF16 almost never overflows or underflows, so you can usually skip loss scaling entirely, whereas FP16 has a narrow window and needs scaling to stay inside it.
Key fact: BF16 was introduced in Google's TPUs and is now native on NVIDIA Ampere (A100) and later, AMD CDNA, and Intel Gaudi. It is the dominant format for large-model pretraining in 2026 precisely because it removes the loss-scaling babysitting that FP16 demands.
The master-weights trick
The optimizer keeps one authoritative FP32 copy of every weight. Each step: cast the FP32 master down to 16-bit, run the forward and backward pass in 16-bit (fast), then apply the resulting update to the FP32 master (precise). Because the master accumulates in 32-bit, a stream of individually-tiny updates still sums correctly over time instead of each one vanishing into rounding. The 16-bit copy is disposable and regenerated every step. This is why "mixed-precision" doubles weight memory for the parameters themselves — you store both copies — yet still wins overall because activations, the memory hog in large batches, are halved.
Loss scaling, concretely
In FP16 the gradients for a healthy network often cluster around 10-4 to 10-8, and anything below ~6×10-8 flushes to zero (denormal underflow). The fix is almost embarrassingly simple: multiply the loss by a large constant S (say 1024 or 65536) before backprop. By the chain rule every gradient is scaled by the same S, shifting the whole distribution up into FP16's representable range. Just before the optimizer updates the master weights, you divide the gradients back by S. Dynamic (automatic) loss scaling goes further: it starts S high, and whenever a gradient overflows to inf/NaN it skips that step and halves S; after a run of clean steps it doubles S again, continuously hunting for the largest safe scale.
Key fact: With BF16 you typically set the loss scale to 1 (i.e. off). Its FP32-equal exponent range means gradients rarely underflow, which is the single biggest reason teams migrated FP16 → BF16 for billion-parameter pretraining.
Where the speed actually comes from
The headline speedups are not magic — they come from dedicated matrix hardware. NVIDIA tensor cores, for example, perform a small matrix multiply-accumulate in one instruction with 16-bit inputs and FP32 accumulation. On an A100 the BF16/FP16 tensor-core throughput is roughly 16× the FP32 CUDA-core rate on paper; real training speedups land around 2–3× end-to-end once you account for memory-bound layers, data loading, and the layers that stay in FP32. Memory traffic halves for activations and gradients, which also lets you fit larger batches or longer sequences. The same principle underlies AMD's Matrix Cores and Intel Gaudi's MME units, and it is why hardware vendors publish their "AI TFLOPS" numbers in BF16/FP16, not FP32.
Not every operation should be demoted. Reductions (softmax, batch-norm statistics, large sums) and the final loss are kept in FP32 because accumulating many small values in 16-bit loses precision catastrophically. Frameworks encode these rules in "autocast" allow/deny lists: torch.cuda.amp.autocast in PyTorch, or mixed_float16 policies in Keras, automatically choose the precision per op so you rarely hand-annotate.
Relation to inference quantization
Mixed-precision training is about speeding up learning while preserving the gradient signal; it is distinct from, but conceptually adjacent to, the aggressive INT8/INT4 schemes used at deployment. Both reduce bit-width to exploit hardware, but inference quantization freezes a trained model and tolerates a one-time accuracy hit, whereas training precision must protect the iterative update dynamics. If you are mapping this onto real silicon, it is worth studying both — see our companion piece on quantization in hardware and the deep dive on tensor cores and systolic arrays to understand the datapaths that make low precision fast.
Teams building custom accelerators or taping out AI silicon often validate these numeric formats in RTL and signoff before committing to a datapath; if you need that kind of design support, Avecas offers VLSI and chip-design services that cover exactly this territory.
Want to go from "I've heard of BF16" to building and profiling a real mixed-precision pipeline on GPU tensor cores? Our CourseTron AI Hardware track walks you through autocast, loss scaling, and kernel-level profiling with worked labs.
FAQ
Do I always need loss scaling?
No. You need it for FP16, where the limited exponent range causes small gradients to underflow. With BF16, which shares FP32's exponent width, the loss scale is usually left at 1 and the mechanism effectively disappears.
Why keep a separate FP32 master copy instead of just rounding in place?
Because weight updates can be far smaller than the weight's own least-significant 16-bit mantissa bit. Accumulating them in FP32 lets many tiny updates sum correctly over thousands of steps; rounding in 16-bit each step would discard them and stall convergence.
How much speedup is realistic?
On tensor-core GPUs expect roughly 2–3× end-to-end wall-clock speedup and about half the activation memory for large models. The theoretical matrix-unit throughput is far higher, but memory-bound and FP32-retained layers cap the real figure.
Is accuracy hurt?
Done correctly, final model accuracy matches FP32 within noise. The master weights, FP32 reductions, and loss scaling together preserve the numerics that matter; the 16-bit compute only affects the bulk matrix multiplies, which are tolerant of reduced mantissa.