In short: A systolic array is a grid of tiny multiply-accumulate (MAC) cells that pass data to their neighbors every clock cycle, so each number fetched from memory is reused across many cells — which is exactly how a Google TPU multiplies large matrices at enormous throughput while keeping memory traffic low.
If you have ever wondered why a GPU is good at matrix math but a TPU is obscene at it, the answer is a 1970s idea from H.T. Kung and Charles Leiserson called the systolic array. The name comes from "systole" — the rhythmic contraction of a heart — because data pulses through the array in lockstep with the clock, cell to cell, without returning to memory in between. For anyone preparing for an AI-hardware or VLSI interview, this is one of the highest-leverage architectures to actually understand, not just name-drop.
The problem: matrix multiply is memory-bound
Consider multiplying a 256×256 matrix by another 256×256 matrix. That is 256³ ≈ 16.7 million MAC operations. The arithmetic itself is cheap — a modern process can place millions of MAC units. The real enemy is data movement. In a naive design, computing one output element C[i][j] means streaming a full row of A and a full column of B from memory. Do that for every output and you re-read the same operands thousands of times. Fetching a byte from DRAM costs roughly 100× the energy of a MAC, so a design that is bottlenecked on fetches wastes almost all its power on moving numbers rather than computing with them.
Key fact: In deep-learning accelerators, energy per DRAM access is on the order of hundreds of times the energy of the arithmetic operation it feeds. Reuse, not raw FLOPS, is the thing good architectures optimize.
The idea: a grid of MAC cells that share data
A systolic array lays out an N×N grid of identical processing elements (PEs). Each PE does one job per cycle: multiply two inputs, add the product to a running sum, and shove its inputs onward to the neighbor PE. Operands enter at the edges and march across the array. Because every PE a value passes through consumes it, a single fetched number does useful work in many cells before it leaves the array. That is the whole trick: amortize one expensive memory read across a row or column of cheap compute.
How the TPU does it: weight-stationary dataflow
Google's first-generation TPU (2015, described in their ISCA 2017 paper) is built around a 256×256 systolic array of 8-bit integer MAC units — 65,536 of them. It uses a weight-stationary scheme: the weights of a neural-network layer are pre-loaded into the PEs and held there, while activations stream in from the left. As each activation crosses a column, it is multiplied by the resident weight and the partial sum ripples downward. Because weights never move once loaded, the array reuses them across every activation in the batch.
At 700 MHz, those 65,536 MACs give the first TPU a peak of about 92 tera-operations per second (65,536 × 2 ops × 700 MHz ≈ 92 TOPS) — all while reading each weight from on-chip memory just once per matrix. That is the payoff of reuse: throughput scales with the N² cells, but off-chip bandwidth scales only with the N edges.
Dataflow variants you should know
Not every systolic array is weight-stationary. The choice of what stays put trades off which operand gets reused and how partial sums are handled. Interviewers love this table:
| Dataflow | What stays in the PE | Reuse maximized | Classic example |
|---|---|---|---|
| Weight-stationary | Weights | Weight reuse across a batch | Google TPU v1 |
| Output-stationary | Accumulator (partial sum) | Partial-sum reuse (no readback) | Many CNN accelerators |
| Row-stationary | A row of filter + activations | All three, balanced | MIT Eyeriss |
Key fact: The array only pays off when it is kept full. A 256×256 array needs ~256 cycles to fill (latency) and ~256 to drain; for small matrices those bubbles dominate, which is why TPUs batch work and why edge NPUs often use smaller 16×16 or 32×32 tiles.
Designing and verifying one in RTL
A systolic array is a joy to build in Verilog precisely because it is one small PE replicated by a generate loop, stitched together with registered neighbor connections. The hard part is not the datapath — it is the timing and the control. Because every PE is purely local (it only ever talks to the cell beside it), the design has short wires and pipelines cleanly, which is why it closes timing at high clock rates in tools like Synopsys Design Compiler or Cadence Genus. Physical implementation in Cadence Innovus or Synopsys IC Compiler II benefits too: the regular, tile-based floorplan maps almost directly onto the layout. For verification, teams lean on Siemens Questa or Synopsys VCS with a reference matrix-multiply model to check that the drained outputs match, cycle-accurately, accounting for the fill/drain skew.
If you want to go from this conceptual view to actually coding and synthesizing one, our hands-on CourseTron VLSI courses walk through the RTL and the timing. For a deeper hardware view of how the surrounding unified buffer, accumulators and control FSM wrap the array, see our companion piece on TPU architecture, and for the broader accelerator landscape, GPU vs TPU for deep learning. Teams building custom silicon around these blocks often partner with specialists like Avecas VLSI design services for tape-out-ready implementation.
Why it matters for AI hardware today
Every major inference accelerator — TPUs, many NPUs in phones, and the matrix engines inside modern GPUs (NVIDIA's Tensor Cores are a close cousin) — is some flavor of this idea: pack MACs densely, keep operands local, and reuse each fetched value as much as possible. Understanding the systolic array gives you the mental model to reason about all of them, and to answer the follow-up questions about utilization, dataflow, and bandwidth that separate a memorized answer from a real one.
Want to build one? CourseTron's AI Hardware track takes you from this diagram to a synthesizable, verified systolic array in Verilog, complete with TPU-style dataflow labs. Explore the course → and train with industry mentors from ChipXpert VLSI training.
Frequently asked questions
Is a systolic array the same as an FPGA or a GPU?
No. A systolic array is a specific dataflow architecture — a grid of locally-connected MAC cells. It can be implemented on an FPGA, as a hard block in an ASIC like the TPU, or emulated on a GPU, but it is the organization of the compute, not the chip technology.
Why 8-bit integers in the TPU instead of floating point?
Inference tolerates low precision. An 8-bit integer MAC is far smaller and lower-power than a 32-bit float MAC, so you fit many more of them in the same area and power budget. 65,536 int8 MACs is only practical because each one is tiny.
What limits a systolic array's efficiency?
Fill and drain latency and poor utilization on odd-shaped matrices. If your matrix is much smaller than the array, most cells sit idle while the pipeline fills and empties, so throughput collapses. Mapping and tiling the problem to keep the array busy is the real engineering challenge.
Who invented the systolic array?
H.T. Kung and Charles Leiserson introduced and named it in 1978–1979 at Carnegie Mellon, originally for signal processing. The deep-learning renaissance — above all Google's TPU — made it famous four decades later.