In short: AI accelerator interconnects like NVIDIA NVLink and AMD Infinity Fabric exist because PCIe is far too slow and too high-latency to let dozens of GPUs behave as one giant accelerator, so vendors build dedicated, switched, cache-coherent fabrics that deliver terabytes per second of intra-node bandwidth and shape multi-GPU topologies around how large-model traffic actually flows.
Why the interconnect became the bottleneck
For a decade, GPU performance was a story about FLOPS. Training a modern large language model flipped that: a single GPU cannot hold a 400-billion-parameter model, so the model is sharded across 8, 72, or thousands of GPUs. The moment you shard, every training step forces the GPUs to exchange gradients and activations. The communication step, not the matrix multiply, now dictates how fast the cluster runs. If the link between accelerators is slow, expensive compute sits idle waiting for data.
PCIe is the obvious candidate and the obvious disappointment. A PCIe 5.0 x16 link gives roughly 64 GB/s per direction. That sounds large until you compare it to a fifth-generation NVLink stack delivering 900 GB/s of bidirectional bandwidth per GPU. A 14x gap, plus PCIe's microsecond-scale latency and its tree topology funnelling everything through a root complex, is why dedicated GPU-to-GPU fabrics were invented.
Key fact: An NVIDIA H100's NVLink provides ~900 GB/s bidirectional bandwidth per GPU versus ~64 GB/s for PCIe 5.0 x16. The interconnect, not the arithmetic unit, is usually the scaling limiter in LLM training.
What NVLink and Infinity Fabric actually are
NVLink is a point-to-point SerDes-based link. Each "link" is a bundle of high-speed differential lanes; a GPU exposes many links that are aggregated into one logical high-bandwidth pipe. Crucially, NVLink is a memory-semantic fabric: a GPU can issue loads and stores directly into a peer GPU's HBM with hardware address translation, rather than packaging a DMA descriptor the way PCIe demands. That removes software overhead and makes collective operations dramatically cheaper.
AMD's Infinity Fabric plays the same role in the Instinct MI300 family and also extends coherence between EPYC CPUs and GPUs, so host and device can share a coherent address space. Both vendors treat the fabric as a first-class design target alongside the compute die — in many recent parts the links are implemented as chiplets stitched together in-package, which is itself a tiny interconnect problem solved with silicon interposers and advanced packaging.
Topology: rings, meshes, and switched fat trees
Bandwidth per link is only half the picture. How the links are wired — the topology — decides whether that bandwidth is usable for a given communication pattern. Three archetypes dominate:
- Ring — each GPU talks to two neighbours. Cheap and the natural fit for ring all-reduce, but latency grows linearly with GPU count and a single slow hop stalls everyone.
- Mesh / hypercube — more direct links reduce hop count, improving latency-sensitive collectives at higher wiring cost.
- Switched fat tree — GPUs connect through dedicated switch chips (NVSwitch), giving non-blocking all-to-all bandwidth. This is why an 8-GPU HGX board or a 72-GPU NVL72 rack behaves almost like one accelerator.
The topology must match the parallelism strategy. Tensor parallelism splits a single layer's matrices across GPUs and hammers the fabric every microsecond, so it belongs inside the highest-bandwidth switched domain. Pipeline and data parallelism tolerate slower links and can span across nodes. We explore how these strategies map to hardware in detail in model parallelism and the hardware it runs on.
Key fact: A ring all-reduce moves 2(N−1)/N times the gradient volume regardless of N, so its cost is bandwidth-bound; a switched fabric collapses the hop count, turning an N-step latency chain into effectively one.
Scaling out: when the fabric leaves the box
NVLink and Infinity Fabric are scale-up fabrics — superb inside a node or rack but not designed to wire a 10,000-GPU data centre. For scale-out, clusters fall back to RDMA networking: InfiniBand or RoCE (RDMA over Converged Ethernet) at 400–800 Gb/s per port, arranged in a fat-tree or dragonfly topology. The art of cluster design is placing tensor-parallel groups entirely inside the fast scale-up domain and letting only the less chatty data-parallel traffic cross the slower scale-out network.
| Fabric | Per-GPU BW (approx) | Scope | Semantics |
|---|---|---|---|
| PCIe 5.0 x16 | 64 GB/s | Host ↔ device | DMA |
| NVLink (gen 5) | ~900 GB/s | Scale-up (node/rack) | Memory-semantic |
| Infinity Fabric | ~800+ GB/s | Scale-up + CPU coherence | Coherent load/store |
| InfiniBand NDR | 400 Gb/s/port | Scale-out (cluster) | RDMA |
How engineers actually build and verify these links
An interconnect is a brutal signal-integrity problem. At 100+ Gb/s per lane the copper trace behaves like a lossy transmission line, so designers lean on PAM4 signalling, forward error correction, and equalisation (CTLE and decision-feedback equalisers) baked into the SerDes PHY. These channels are modelled and closed with tools such as Synopsys HSPICE and Cadence Sigrity/Clarity for the channel, while the digital controller and link-layer logic are verified in SystemVerilog/UVM environments (Synopsys VCS, Cadence Xcelium, Siemens Questa) long before silicon exists. Protocol correctness — credit-based flow control, retry on CRC error, lane reversal — is exactly the kind of corner-case behaviour that formal and constrained-random verification exist to catch. Teams designing custom accelerators and their fabrics often partner with specialists like Avecas VLSI design services for this physical-and-verification heavy lifting.
If you want to go deeper on the die-to-die and packaging side that increasingly defines these fabrics, see our companion piece on chiplets and die-to-die interconnect.
Want to build this intuition hands-on? CourseTron's AI Hardware track walks through interconnect topologies, collective communication, and the RTL-to-signoff flow behind real fabrics. Explore the full catalogue on our courses page.
Frequently asked questions
Is NVLink faster than PCIe, and by how much?
Yes — roughly an order of magnitude. A gen-5 NVLink stack offers about 900 GB/s bidirectional per GPU against ~64 GB/s for PCIe 5.0 x16, and it uses memory semantics so peer GPUs can read each other's HBM directly with far lower latency.
Why not just use faster Ethernet everywhere instead of NVLink?
Ethernet/RDMA is excellent for scale-out across thousands of nodes, but it cannot match the sub-microsecond latency and memory-semantic access of an in-package or in-rack fabric. Tensor-parallel traffic is too chatty and latency-sensitive to tolerate a network hop, so the two coexist.
What is a switched topology and why does it matter?
A switched topology routes GPU traffic through dedicated switch chips (e.g. NVSwitch) so every GPU can reach every other at full bandwidth without blocking. This makes all-reduce and all-to-all collectives — the backbone of distributed training — scale cleanly instead of degrading with GPU count.
Does the interconnect limit inference too, or only training?
Both. Large-model inference uses tensor parallelism to fit weights across GPUs and to hit latency targets, so token generation also depends on fast GPU-to-GPU links; the KV-cache and activation exchanges ride the same fabric.