In short: LLM token generation is bottlenecked by memory bandwidth, not math, because every new token must re-read the entire growing key/value (KV) cache from HBM — so the real wins come from shrinking and managing that cache through paging, quantization, and batching rather than from faster FLOPs.
Ask any engineer why their 70-billion-parameter chatbot feels slow and they will usually blame "not enough compute." For the decode phase of a transformer, that is almost always the wrong diagnosis. Decoding is overwhelmingly memory-bound, and the single biggest consumer of that memory traffic is the KV cache. Understanding it is now a core interview topic for AI-hardware and inference roles, and it ties directly into the datapath concerns we cover in transformer hardware acceleration.
Why attention keeps a cache at all
Self-attention lets every token look back at all previous tokens. For a given token, the model projects the hidden state into a query (Q), a key (K), and a value (V) vector. The attention score for position i is a softmax over the dot products of its Q with the K of every earlier position, and the output is the weighted sum of those V vectors.
During autoregressive generation we emit one token at a time. The crucial observation: the K and V of token 5 never change when we generate token 6, 7, or 800. Recomputing them every step would be O(n²) wasted work. So we cache K and V for all past tokens and only compute Q, K, V for the single newest token. That cache is the KV cache.
The memory-bound arithmetic
Here is where the hardware story bites. The decode step does very little arithmetic per byte it touches. Multiplying the new Q against the cached K matrix is a tall-skinny GEMV, not a fat GEMM. The arithmetic intensity (FLOPs per byte moved) is roughly 1, while a modern GPU like an NVIDIA H100 wants about 300 FLOPs/byte to keep its tensor cores busy. The result: the SMs sit idle waiting on HBM.
Concretely, size the cache. Per token per layer you store K and V: 2 × n_heads × head_dim × bytes. For Llama-2-13B (40 layers, hidden 5120) in FP16 that is about 800 KB per token. A single 4,000-token context therefore needs roughly 3.2 GB of KV cache — for one request. Every decode step must stream that whole structure through the memory system.
At 2,000-token context, a 13B model's KV cache can dwarf the activations and rival a meaningful fraction of the weights. This is why doubling batch size often doubles latency: you are bandwidth-limited, not FLOP-limited.
| Model | Layers | KB / token | 4K-ctx cache |
|---|---|---|---|
| Llama-2-7B | 32 | ~512 | ~2.0 GB |
| Llama-2-13B | 40 | ~800 | ~3.2 GB |
| Llama-2-70B (GQA) | 80 | ~320 | ~1.3 GB |
Paging: PagedAttention and vLLM
The naive implementation reserves one contiguous buffer sized to the maximum sequence length for every request. If a request only emits 50 tokens but you reserved 4,096, you waste 98% of that memory, and fragmentation means you cannot pack more concurrent requests. Measured internal fragmentation in early serving stacks exceeded 60–80%.
PagedAttention, introduced with the vLLM serving engine, borrows the idea of virtual memory. The KV cache is split into fixed-size blocks (say 16 tokens each), and a per-request block table maps logical positions to physical blocks scattered anywhere in GPU memory. Blocks are allocated on demand, so a 50-token request uses four blocks, not 256. This alone lifts achievable batch sizes several-fold and lets identical prompt prefixes be shared copy-on-write across requests.
Quantized KV cache
If the cache is the bandwidth problem, store it in fewer bits. The weights of production models are routinely quantized to INT8 or INT4; the KV cache can be too. Dropping K and V from FP16 to INT8 halves the bytes moved and roughly halves decode latency in the bandwidth-bound regime, with per-channel or per-token scaling keeping accuracy loss under a point of perplexity for most models. INT4 KV (used in schemes like KVQuant and KIVI) can cut it again, though keys are more sensitive than values and usually need finer-grained scales.
A complementary architectural fix is Grouped-Query Attention (GQA) and its extreme, Multi-Query Attention (MQA): instead of one K/V pair per query head, several query heads share one K/V head. Llama-2-70B uses 8 KV heads for 64 query heads, shrinking its cache 8× versus full multi-head attention — the reason the 70B row above is cheaper than 13B.
Batching to amortize the bandwidth
Because weights are read from HBM once per layer regardless of how many sequences you process, batching many requests together spreads that fixed cost. Continuous (in-flight) batching goes further: rather than waiting for the slowest sequence in a static batch to finish, the scheduler adds and retires sequences every step, keeping the GPU saturated. Combined with paging, this is what turns a single H100 from serving a handful of users to serving dozens.
Rule of thumb: in the memory-bound decode regime, throughput scales almost linearly with batch size until the KV cache fills HBM — at which point paging and quantization buy you the headroom to keep scaling.
These same memory-hierarchy principles show up across the stack; if you are also studying how data movement dominates accelerator design, see our piece on memory hierarchy in AI accelerators. For hands-on RTL and verification skills behind custom inference silicon, our partners at Avecas VLSI design services build and tape out exactly this class of datapath.
Want to go from "I've heard of KV cache" to designing and profiling your own inference engine? CourseTron's AI Hardware track walks through PagedAttention, quantization, and roofline analysis with labs. Browse the full course catalog.
Frequently asked questions
Is LLM prefill also memory-bound?
No. Prefill processes the whole prompt in parallel, which is a large matrix–matrix multiply with high arithmetic intensity, so it is compute-bound. Only the one-token-at-a-time decode phase is memory-bound, which is why the two phases are often scheduled and even run on different hardware.
Does quantizing the KV cache hurt quality?
Modestly. INT8 KV with per-token scaling is nearly lossless for most models; INT4 is riskier, especially for keys, and benefits from outlier-aware or per-channel schemes. Always validate perplexity and a few downstream tasks before shipping.
How is this different from quantizing the weights?
Weight quantization shrinks the static model stored once; KV quantization shrinks a per-request structure that grows with context length. They are orthogonal and are usually applied together.
Which tools help me profile this?
NVIDIA Nsight Systems and the roofline view in Nsight Compute reveal whether a kernel is bandwidth- or compute-bound. On the silicon side, EDA flows from Synopsys, Cadence, and Siemens EDA are used to design the HBM controllers and on-chip SRAM that set these bandwidth limits.