Model/tensor/pipeline parallelism

In short: When a model is too large for one accelerator's memory or too slow on one chip, you split it across many devices using tensor parallelism (slice each layer), pipeline parallelism (split the layer stack into stages), and data parallelism (replicate and shard optimizer state) — and the interconnect, not the FLOPs, usually decides whether it scales.

A 70-billion-parameter transformer in FP16 needs roughly 140 GB just to hold weights, and training it needs several times that for gradients and optimizer state. No single accelerator — not an 80 GB NVIDIA H100, not a 192 GB MI300X — fits the whole job. The answer is to spread one model across tens, hundreds, or thousands of chips. How you spread it, and how fast those chips can talk, is the entire ballgame.

The three axes of parallelism

Modern large-model systems compose three orthogonal strategies. Understanding where each one cuts the computation is the single most useful mental model you can build.

• Data parallelism (DP): every device holds a full copy of the model and processes a different slice of the batch. Gradients are averaged with an all-reduce. Simple, but it does nothing for a model that won't fit in one device — each replica still needs the whole model.
• Tensor parallelism (TP): a single layer's matrix multiply is split across devices. A weight matrix of shape [H, 4H] in a feed-forward block is column-partitioned so each of 4 GPUs computes a quarter, then the partial results are combined. This shrinks per-device memory but demands an all-reduce or all-gather inside every layer, on the critical path.
• Pipeline parallelism (PP): the stack of layers is split into sequential stages, each on a different device. Layers 1–10 live on GPU 0, 11–20 on GPU 1, and activations are handed forward like a factory line. Communication is small (just activations between stages) but you pay a "bubble" of idle time while the pipeline fills and drains.

Rule of thumb: tensor parallelism is the most bandwidth-hungry and is normally kept inside a single node (over NVLink/NVSwitch). Pipeline and data parallelism tolerate the slower inter-node fabric (InfiniBand/Ethernet). Getting this placement wrong can cut throughput by more than half.

Why the interconnect decides everything

A tensor-parallel all-reduce moves data proportional to the hidden dimension and batch, and it happens twice per transformer layer. For a model with 80 layers that is 160 collective operations per forward-backward step — each one a hard synchronization barrier. If your link is slow, every GPU waits.

This is why topology matters. Within an NVIDIA DGX H100 node, NVLink 4 + NVSwitch delivers about 900 GB/s of all-to-all bandwidth per GPU. Step outside the node onto InfiniBand NDR and you drop to roughly 400 Gb/s (~50 GB/s) per link — nearly 18× slower. Put a tensor-parallel group across that boundary and the all-reduces stall the whole training run. Keep TP inside the node and PP/DP across nodes, and the fast links carry the chatty traffic while the slow links carry only occasional activation handoffs.

Taming the pipeline bubble

Naive pipelining wastes time: with 4 stages, 3 GPUs sit idle while stage 0 processes the first micro-batch. The fix is to chop the batch into many micro-batches and keep the pipeline full. GPipe and the 1F1B (one-forward-one-backward) schedule used in Megatron-LM reduce the bubble fraction to roughly (stages − 1) / micro-batches. With 8 stages and 64 micro-batches the bubble is about 11% — acceptable. Interleaved schedules push it lower still by giving each device several non-contiguous stages.

Concrete example: training a 530B-parameter model (Megatron-Turing NLG) used 8-way tensor, 35-way pipeline, and data parallelism together across 2,240 A100 GPUs — a "3D parallelism" mesh. No single axis could have done it alone.

Memory tricks that complement sharding

Parallelism splits compute; you also fight memory directly. ZeRO (used in DeepSpeed and PyTorch FSDP) shards optimizer states, gradients, and parameters across data-parallel ranks instead of replicating them — cutting per-device memory by the DP degree while keeping the simplicity of data parallelism. Activation checkpointing trades compute for memory by recomputing activations in the backward pass rather than storing them, often saving 60–70% of activation memory at a ~30% compute cost. These stack on top of TP and PP.

Where silicon design meets the software mesh

None of this is purely a software problem. The accelerators, the NVSwitch fabric, and the network interface cards are all chips that must be verified, laid out, and signed off. Teams building AI accelerators and high-bandwidth interconnect rely on EDA flows — Synopsys and Cadence for RTL-to-GDS and high-speed SerDes design, Siemens EDA (Calibre) for physical verification of these enormous dies — the same back-end discipline we teach in our physical-design track. If you are weighing where a workload belongs, it is worth comparing this many-chip approach with the single-giant-die philosophy of the wafer-scale engine, and with how high-bandwidth memory feeds these accelerators. For hands-on chip design services behind such accelerators, Avecas's VLSI design team works these flows in production.

Want to go from "I've heard of NVLink" to actually reasoning about 3D parallelism meshes? Our AI Hardware & Accelerator Systems course on CourseTron walks through interconnect topology, collective communication, and real training-run layouts with worked examples.

Frequently asked questions

When should I use tensor parallelism instead of just more data parallelism?

Use tensor parallelism when a single layer's activations and weights no longer fit comfortably in one device, or when you need to cut per-device memory below what ZeRO/FSDP alone achieves. Data parallelism helps throughput but never reduces the model copy each device must hold.

Why is pipeline parallelism considered "cheap" on communication?

It only passes activation tensors between adjacent stages — point-to-point, once per micro-batch per boundary — whereas tensor parallelism performs bandwidth-heavy all-reduces inside every layer. That is why pipeline cuts can safely cross the slower inter-node fabric.

What is the "bubble" and how do I minimize it?

The bubble is the idle time while a pipeline fills and drains. Increase the number of micro-batches relative to the number of stages, and use a 1F1B or interleaved schedule, to drive the bubble fraction toward a few percent.

Does the interconnect really matter more than raw FLOPS?

For large-model training, very often yes. A GPU that waits on a slow all-reduce delivers little of its peak FLOPS. Fabric bandwidth and topology-aware placement routinely determine whether a cluster scales linearly or stalls.