In short: Modern deep-learning hardware is bottlenecked not by raw math but by memory bandwidth — the "memory wall" — so engineers use the roofline model to decide whether a kernel is compute-bound or memory-bound, and pick training chips (high-bandwidth, high-precision, interconnect-rich) versus inference chips (low-latency, low-precision, energy-frugal) accordingly.
Ask most students "what makes a GPU fast for AI?" and they will say TFLOPS. That answer is incomplete and often wrong. A chip can advertise a petaflop of compute and still crawl, because the multiply-accumulate units sit idle waiting for data. Understanding why is the single most useful mental model in AI hardware — and it is exactly what interviewers at Nvidia, AMD, Google and the design-services houses probe for.
The Memory Wall: Why FLOPS Lie
Over two decades, peak arithmetic throughput on accelerators grew roughly 60,000x, while DRAM bandwidth grew only about 100x and interconnect bandwidth even less. That widening gap is the memory wall. The consequence: for many layers, the processor finishes its arithmetic long before the next operands arrive from HBM (high-bandwidth memory).
Consider a concrete example. An Nvidia A100 delivers roughly 312 TFLOPS of FP16 tensor math but only about 2,039 GB/s of HBM2e bandwidth. To keep the tensor cores busy, every byte fetched must feed roughly 150 floating-point operations. A dense matrix multiply of large square matrices reuses data heavily and easily clears that bar. A vector add (one multiply per two loads) does not come close — it runs at a tiny fraction of peak, limited purely by bandwidth.
Rule of thumb: if a kernel does fewer FLOPs per byte than the machine's "ops-per-byte" balance point, it is memory-bound and buying more compute will not help it at all.
The Roofline Model: One Chart That Explains Everything
The roofline model, formalized by Williams, Waterman and Patterson at Berkeley, plots attainable performance (GFLOP/s, log scale) against arithmetic intensity (FLOPs per byte of DRAM traffic). It produces two ceilings: a slanted line set by memory bandwidth and a flat line set by peak compute. Any kernel is a point under that roof.
The ridge point — where the slope meets the flat roof — is the arithmetic intensity your workload must exceed to be worth the chip's full compute. For the A100 that is about 150 FLOPs/byte. This is why operator fusion matters so much: FlashAttention famously rewrote attention to avoid writing the huge intermediate score matrix to HBM, raising its arithmetic intensity and moving the kernel from memory-bound to compute-bound without changing the math.
Watching the Wall in Motion
The animation below shows the core drama: a wide multiply-accumulate (MAC) array that can consume data every cycle, fed by a single narrow pipe from memory. When the pipe can't keep up, the array stalls.
Training vs Inference: Two Different Machines
Training and inference stress hardware along opposite axes. Training does forward and backward passes, stores activations for gradients, and synchronizes weights across thousands of chips — it is bandwidth-, memory-capacity- and interconnect-bound. Inference runs forward-only on one request (or a small batch), where latency and energy per token dominate, and precision can be aggressively reduced.
| Dimension | Training hardware | Inference hardware |
|---|---|---|
| Dominant metric | Throughput, scaling efficiency | Latency, tokens/joule |
| Numeric precision | BF16 / FP16, FP8 emerging | INT8 / INT4 / FP8 |
| Memory need | Huge (weights + activations + optimizer state) | Weights + KV cache only |
| Interconnect | Critical (NVLink, InfiniBand, TPU ICI) | Often single-chip |
| Examples | H100/B200, TPU v5p, MI300X | L4, Inferentia, edge NPUs |
An INT8 multiply costs roughly one-quarter the energy and area of an FP16 multiply, and INT4 less still. That is why inference accelerators lean hard into quantization — but it is also why they are useless for training, where low precision destroys gradient signal.
During autoregressive LLM inference a second wall appears: the KV cache. Every generated token must read the entire growing key/value cache from memory, so token generation is almost purely memory-bandwidth-bound — which is precisely why batching and techniques like paged attention exist, and why inference silicon prioritizes bandwidth-per-watt over peak FLOPS. For a deeper split of silicon families, see our companion piece on GPU vs TPU for AI.
How This Shows Up in Real Chip Design
These ideas are not just for buyers — they drive the RTL and physical design of the accelerators themselves. Dataflow architectures (systolic arrays like the TPU's MXU) are deliberately engineered to maximize operand reuse so each HBM byte feeds many MACs. Verifying such designs leans on tools like Synopsys VCS and Siemens Questa for simulation, while physical implementation and power/timing closure use Synopsys Fusion Compiler or Cadence Innovus. Memory-system choices — SRAM banking, HBM stacks on a silicon interposer, network-on-chip topology — are where most of the real engineering effort goes. Teams building custom AI silicon often partner with design-services specialists such as Avecas VLSI design services to tape out these interconnect-heavy blocks. If you want the quantization and dataflow angle in more depth, read our note on quantization for AI inference.
The takeaway for anyone entering the field: optimize what is actually limiting you. Profile first, locate the kernel on the roofline, then decide whether the fix is more compute, more bandwidth, better data reuse, or lower precision. Guessing wastes silicon and salary.
Want to go from "I've heard of HBM" to designing memory hierarchies and running real EDA flows? Explore the AI Hardware and VLSI tracks in the CourseTron course catalogue.
Frequently Asked Questions
What is the memory wall in simple terms?
It is the growing gap between how fast a processor can compute and how fast it can fetch data from memory. Compute has improved far faster than memory bandwidth, so chips increasingly sit idle waiting for operands rather than running out of math.
How do I know if my deep-learning kernel is memory-bound?
Compute its arithmetic intensity (FLOPs divided by bytes moved from DRAM) and compare it to the hardware's ridge point — about 150 FLOPs/byte on an A100. Below the ridge you are memory-bound; profilers like Nsight Compute will confirm by showing high memory utilization and low tensor-core occupancy.
Why can't I use the same chip for training and inference?
You can, but it is inefficient. Training needs high precision, enormous memory for activations and optimizer state, and fast chip-to-chip interconnect. Inference wants low precision, low latency and minimal energy per token. Purpose-built inference chips strip out what training needs and win dramatically on cost per query.
Does FP8 change the training-versus-inference picture?
Yes, partly. FP8 (with per-tensor scaling) now lets some training happen at 8-bit, narrowing the gap. But training still needs higher-precision accumulation and far more memory and interconnect, so the two hardware classes remain distinct.