In short: From Volta to Blackwell, NVIDIA kept the same recipe, a faster matrix engine fed by faster memory, but every generation removed a different bottleneck: Volta introduced tensor cores, Ampere added sparsity and flexible formats, Hopper added FP8 with the Transformer Engine, and Blackwell went to FP4 and a dual-die package.
The one idea behind every generation
Interviewers rarely ask you to recite specs. They ask why a generation looked the way it did. The answer is almost always a roofline argument. A GPU has a peak math rate (FLOP/s) and a peak memory rate (bytes/s). Their ratio is the ridge point: the arithmetic intensity a kernel needs before it stops waiting on memory. When tensor cores get faster than HBM gets wider, the ridge point climbs and more of your model becomes memory-bound. Each architecture below is a response to that tension.
Key fact: V100 offers about 125 TFLOPS of FP16 tensor math against 0.9 TB/s of HBM2, a ridge near 139 FLOP/byte. H100 SXM offers roughly 989 TFLOPS dense FP16 against 3.35 TB/s, a ridge near 295. Math outran memory by 2x in five years.
Volta: the tensor core is born
Before Volta, a CUDA core did one fused multiply-add per clock. Volta's V100 (12 nm, 21 billion transistors, 815 mm²) added 640 tensor cores, each computing a 4x4x4 matrix multiply-accumulate per clock: D = A x B + C, with FP16 inputs and FP32 accumulation. That is 64 FMAs in one instruction, which is why mixed-precision training jumped to about 125 TFLOPS. Keeping the accumulator in FP32 matters: summing thousands of FP16 products in FP16 loses small gradients. Volta also introduced independent thread scheduling, so threads in a warp have their own program counters. That fixed intra-warp deadlocks but made warp-synchronous code unsafe, hence __syncwarp(). For a deeper dive on the unit itself, read our tensor core explainer.
Ampere: formats and sparsity
Training in FP16 needs loss scaling because its 5-bit exponent overflows or underflows easily. Ampere's A100 (7 nm, 54 billion transistors) added BF16, which keeps FP32's 8-bit exponent with a 7-bit mantissa, and TF32, a 19-bit format that lets unmodified FP32 code use tensor cores. Dense FP16 doubled to 312 TFLOPS. Ampere also added 2:4 structured sparsity: in every group of four weights, two are zero, and the hardware skips them for up to 2x throughput. Memory moved to 40 or 80 GB of HBM2e at 1.55 to 2.0 TB/s, and asynchronous global-to-shared copies (cp.async) let loads bypass registers and overlap with compute. MIG split one A100 into up to seven isolated instances for inference.
Key fact: BF16 versus FP16 is an exponent-versus-mantissa trade. BF16 has 8 exponent and 7 mantissa bits; FP16 has 5 and 10. Range matters more than precision for gradients, so BF16 removed most loss-scaling headaches.
Hopper: FP8 and the Transformer Engine
By 2022 transformers dominated, and their cost was matrix multiplies plus attention memory traffic. H100 (TSMC 4N, 80 billion transistors) answered with three changes.
- FP8 tensor cores in two flavours, E4M3 (more precision, for weights and activations) and E5M2 (more range, for gradients). Dense FP8 reaches about 1,979 TFLOPS, double FP16.
- The Transformer Engine, a hardware plus library pairing that tracks per-tensor amax statistics over recent iterations, picks a scale factor, and casts each layer's tensors to FP8 or leaves them in 16-bit. Scaling is what stops FP8's tiny range from clipping.
- Data movement: the Tensor Memory Accelerator (TMA) copies whole tiles asynchronously, thread block clusters let neighbouring SMs share distributed shared memory, and
wgmmalets a warpgroup of 128 threads issue one large matrix op.
Memory jumped to HBM3 at 3.35 TB/s with 80 GB, and NVLink 4 to 900 GB/s. The H200 kept the same compute but moved to HBM3e: 141 GB at 4.8 TB/s, which sped up LLM inference by roughly 1.4x to 1.9x purely because decoding is memory-bound.
Blackwell: FP4, two dies, tensor memory
Hopper's die was already near the reticle limit (about 800 mm²), so the next doubling could not come from a bigger chip. B200 (TSMC 4NP, 208 billion transistors) instead joins two reticle-sized dies with a 10 TB/s on-package link (NV-HBI), presenting them to software as one GPU. Fifth-generation tensor cores add FP6 and FP4, plus micro-scaling formats where small blocks of 16 or 32 values share a scale. That gives roughly 9 PFLOPS dense FP4 per B200, and the second-generation Transformer Engine automates the choice. The cores also use a dedicated tensor memory (TMEM) for accumulators, freeing registers, and a single thread can issue MMA instructions. Memory is HBM3e at about 8 TB/s and 192 GB, and NVLink 5 reaches 1.8 TB/s per GPU, which is what makes 72-GPU NVL72 racks behave like one accelerator.
Generation comparison
| GPU | Memory | Bandwidth | New low precision | Main fix |
|---|---|---|---|---|
| V100 | 16/32 GB HBM2 | 0.9 TB/s | FP16 (FP32 acc) | Dedicated matrix unit |
| A100 | 40/80 GB HBM2e | 1.55-2.0 TB/s | TF32, BF16, 2:4 sparse | Range and sparsity |
| H100 | 80 GB HBM3 | 3.35 TB/s | FP8 | Transformer throughput |
| B200 | 192 GB HBM3e | ~8 TB/s | FP4, FP6 | Reticle limit, inference cost |
Seeing the memory wall in motion
The animation shows why bandwidth, not FLOPS, often sets inference speed. Weights stream from HBM through L2 into the SM, where the tensor core consumes them. If the pipe is slower than the core, the core idles.
What this means for chip designers
These choices cascade into implementation. HBM stacks sit on a silicon interposer (TSMC CoWoS), so package signal integrity and power delivery dominate, and teams use Cadence Sigrity or Synopsys 3DIC Compiler and Ansys RedHawk for interposer and IR-drop work. Dense MAC arrays at FP4 and FP8 are an exercise in timing closure and clock gating, typically run through Synopsys Fusion Compiler or Cadence Innovus, with Siemens Calibre for physical verification. If you want a design-services view of AI accelerator back-end, see Avecas VLSI design services. To learn the foundations behind these blocks, explore our HBM memory architecture guide and the CourseTron course catalogue.
Key fact: Each halving of precision doubles tensor throughput per transistor and halves memory traffic per weight, so FP16 to FP8 to FP4 helps compute and bandwidth at once. The cost is accuracy engineering: scaling, calibration, and quantization-aware training.
Ready to go deeper? Join the CourseTron AI hardware course to learn tensor cores, memory hierarchies and accelerator design with hands-on labs. Browse courses.
Frequently asked questions
Why did NVIDIA move from FP16 to FP8 and FP4?
Neural networks tolerate low precision, and each halving doubles math throughput and halves memory traffic. Scaling and FP32 accumulation preserve accuracy.
What does the Transformer Engine actually do?
It selects FP8 or 16-bit per layer, tracks tensor maxima to set scale factors, and handles casts automatically so frameworks need minimal code changes.
Why is H200 faster than H100 with the same compute?
LLM decoding is memory-bound. H200's HBM3e raises capacity to 141 GB and bandwidth to 4.8 TB/s, so weights and KV cache stream faster.
Why does Blackwell use two dies?
A single die cannot grow past the reticle limit of roughly 800 mm². Two dies joined by a 10 TB/s link act as one logical GPU.