In short: HBM stacks several DRAM dies vertically, wires them with thousands of through-silicon vias, and places the whole cube next to a GPU or AI accelerator on a silicon interposer — trading a narrow, fast off-package bus for a very wide, moderate-clock one that delivers terabytes per second at far lower energy per bit.
Why ordinary DRAM ran out of road
For two decades, memory bandwidth scaled by pushing the DDR/GDDR interface faster: wider prefetch, higher data rates, equalisation tricks on the PCB. That approach hits a wall set by physics. A GDDR6 chip signals at 16–20 Gb/s per pin, but every one of those pins is a long, lossy trace running across a package and board to a DRAM package centimetres away. Driving those lines at multi-gigahertz edges burns real energy — roughly 15–20 picojoules per bit — and the routing congestion caps how many you can realistically fan out. A modern training GPU needs several terabytes per second; you simply cannot solder enough GDDR packages around a die to get there.
High Bandwidth Memory solves the problem from the opposite direction. Instead of a narrow-and-fast bus, HBM uses a wide-and-near one: a 1024-bit interface per stack running at a comparatively relaxed per-pin rate, with the memory sitting micrometres from the processor. Width replaces frequency, and distance collapses.
Numbers that matter: One HBM3 stack offers a 1024-bit data bus. At ~6.4 Gb/s per pin that is ~819 GB/s per stack. Six stacks around a GPU (a common AI configuration) deliver ~4.9 TB/s — an order of magnitude beyond what a GDDR board can sustain, at roughly 3–5 pJ/bit.
The 3D stack: dies, TSVs and microbumps
An HBM device is a cube of 4, 8, 12 or 16 DRAM dies stacked on top of a logic base die. The dies are thinned to tens of microns and bonded, then connected vertically by through-silicon vias (TSVs) — copper columns etched straight through the silicon so a signal can travel from the top die to the base without ever leaving the stack. Between adjacent dies sit microbumps (µbumps) on a pitch of around 40–55 µm, the solder joints that actually carry each TSV connection from one die to the next.
The base die at the bottom is the traffic cop. It aggregates the per-die channels, handles the PHY-side signalling to the host, and in newer generations hosts test and repair logic. HBM internally splits into independent channels (16 pseudo-channels in HBM3), so the host can keep many banks busy in parallel — this concurrency, not just raw width, is what feeds a GPU's thousands of threads.
The interposer: why the memory has to be on-package
A thousand signals cannot fan out on an ordinary PCB — the trace pitch is too coarse. HBM therefore rides on a silicon interposer (the 2.5D approach, branded CoWoS by TSMC or EMIB by Intel). The interposer is itself a piece of silicon with micron-scale redistribution layers, so you can route the full 1024-bit bus from the stack to the GPU over a span of a few millimetres. Short, dense, low-capacitance wiring is exactly what lets HBM clock slowly yet move enormous bandwidth cheaply. EMIB is a variant that embeds a small silicon bridge only under the die-to-die boundary, saving the cost of a full interposer.
Design-tool reality: HBM integration is a 2.5D packaging problem handled in tools like Cadence Integrity 3D-IC and Siemens Innovator3D IC for planning/co-design, with signal- and power-integrity closure in Cadence Clarity / Sigrity and Synopsys HSPICE. Thermal is first-class: a 16-high stack traps heat under the GPU, so die-stack thermal co-simulation is mandatory, not optional.
Generations and the bandwidth curve
| Generation | Per-pin rate | Bus width | Peak BW/stack | Max dies |
|---|---|---|---|---|
| HBM2 | ~2.0 Gb/s | 1024-bit | ~256 GB/s | 8 |
| HBM2E | ~3.6 Gb/s | 1024-bit | ~460 GB/s | 8–12 |
| HBM3 | ~6.4 Gb/s | 1024-bit | ~819 GB/s | 12–16 |
| HBM3E | ~9.2 Gb/s | 1024-bit | ~1.2 TB/s | 12–16 |
The pattern is instructive: width stays constant at 1024 bits while JEDEC raises the per-pin rate and die count each generation. HBM4, defined by JEDEC in 2025, breaks that rule by doubling the interface to 2048 bits per stack — the first width change since HBM began, acknowledging that pin-rate scaling alone was slowing.
A worked example: feeding an AI training run
Consider a large transformer in mixed precision. During training you repeatedly stream weights, activations and gradients between compute and memory; for many layers the kernel is memory-bound, meaning the GPU's math units stall waiting for operands. If a layer must read 40 GB of tensors per step and your memory delivers 4.9 TB/s, that traffic alone costs ~8 ms; drop to a 1 TB/s GDDR-class subsystem and the same step takes ~40 ms. Bandwidth, not FLOPS, sets the wall-clock. This is precisely why every serious AI accelerator — NVIDIA H100/H200, AMD Instinct MI300, and custom silicon — commits to HBM despite its cost and packaging complexity. If you are building or verifying such datapaths, the die-to-die plumbing that connects these stacks is covered in our companion piece on UCIe chiplet interconnect, and the memory-access ordering rules in AXI protocol fundamentals.
Reliability: repair and RAS built in
Thousands of TSVs and microbumps mean some will fail — in manufacturing or over life. HBM builds this in: spare TSVs and redundant rows allow soft-repair, and HBM3 adds on-die ECC plus link-level features so a single marginal via does not scrap a $-thousands stack. The base die runs built-in self-test (MBIST) to find and remap faults. For an AI farm running months-long jobs, this RAS story is as important as the headline bandwidth.
Where HBM fits in a system engineer's mental model
- HBM is capacity-limited, not bandwidth-limited. A stack tops out at tens of GB, so system DRAM (DDR5/LPDDR) still holds the bulk dataset while HBM is the hot working set.
- It is a packaging decision as much as a memory one. Choosing HBM commits you to 2.5D integration, interposer cost, KGD (known-good-die) sourcing and thermal design — not just a BOM line.
- Latency is comparable to DDR, not better. HBM wins on throughput and energy, not on single-access latency; architects exploit it with massive parallelism.
For teams moving from theory to tapeout, hands-on 2.5D/3D flow experience is what separates an interview answer from a real skill — the kind of physical-design and packaging work our partner Avecas VLSI design services runs on production nodes.
Want to master memory subsystems, interposers and the AI-accelerator datapath end to end? Explore the VLSI and advanced-packaging tracks on our CourseTron courses page.
FAQ
Why is HBM more energy-efficient than GDDR?
Because the memory sits micrometres away on a silicon interposer instead of centimetres away on a PCB, the signalling lines are short and low-capacitance. HBM can therefore clock each pin slowly (a few Gb/s) yet move huge aggregate bandwidth, spending roughly 3–5 pJ/bit versus 15–20 pJ/bit for long, fast GDDR traces.
What exactly is a TSV?
A through-silicon via is a copper-filled hole etched vertically through a thinned DRAM die, letting a signal pass straight up the stack from one die to the next without routing around the edge. Thousands of TSVs give HBM its 1024-bit vertical bandwidth.
Do I always need a silicon interposer for HBM?
You need some high-density 2.5D routing. The classic option is a full silicon interposer (TSMC CoWoS); Intel's EMIB embeds a smaller silicon bridge only under the die-to-die seam. Either way you cannot fan out 1024+ signals on an ordinary organic substrate.
How does HBM4 differ from HBM3E?
HBM4 doubles the per-stack interface to 2048 bits — the first width change in the standard — rather than relying solely on higher pin rates. It targets well over 1.5 TB/s per stack and tighter logic-base-die customisation for specific accelerators.