Dataflow architectures for AI

In short: A dataflow architecture decides which operand — weights, partial sums, or input activations — is held still inside each processing element so the same bits are reused many times, because moving a byte out to DRAM costs roughly 100–200× the energy of the multiply-accumulate that consumes it.

Modern deep-learning accelerators do not live or die by how many multiply-accumulate (MAC) units they pack onto a die. They live or die by how little data they move. In a convolution or a matrix multiply, every operand can be reused dozens or hundreds of times, and a good dataflow is the scheduling discipline that captures that reuse inside cheap, nearby storage instead of paying the brutal tax of off-chip memory. This article unpacks the three canonical dataflows — weight stationary, output stationary, and row stationary — explains the energy arithmetic that makes them matter, and shows how they map onto real silicon.

Why data movement, not arithmetic, is the enemy

The classic numbers from Horowitz and the MIT Eyeriss group frame the whole field. On a 45 nm process, a 32-bit integer add costs about 0.1 pJ and a 32-bit MAC a few pJ. Reading the same 32-bit operand from a small local register file near the ALU costs around 1 pJ; from a 100 KB on-chip SRAM, roughly 5 pJ; and from off-chip DRAM, on the order of 640 pJ. That is a 100–200× gap between the compute and the fetch that feeds it.

A single DRAM access can cost more energy than 200 MACs. If each weight you pull from DRAM is used only once, memory energy dominates the entire chip. Dataflow is the art of using every fetched byte as many times as possible before letting it go.

The implication is stark: an accelerator can have a theoretical peak of 100 TOPS and still deliver terrible performance-per-watt if its dataflow forces it to re-fetch operands from DRAM. The job of the architect is to build a memory hierarchy — DRAM → global SRAM buffer → inter-PE network → per-PE register file — and then choose a dataflow that keeps hot data in the cheapest tier.

Reuse: the three dimensions you can exploit

In a convolutional layer there are three distinct forms of data reuse, and each dataflow is essentially a bet on which one to harvest first:

• Weight reuse (convolutional reuse): the same filter weight slides across every spatial position of the input feature map.
• Input reuse (feature-map reuse): the same input activation feeds many different output channels — i.e. many different filters.
• Output / partial-sum reuse: a single output pixel accumulates contributions from the whole filter window and all input channels, so its partial sum is read-modified-written many times.

You cannot hold all three still at once, so the dataflow names whichever operand is pinned in the PE's register file.

Weight stationary (WS)

In a weight-stationary dataflow each PE loads one weight into its local register and keeps it there while a stream of input activations flows past. Partial sums are passed between PEs and accumulated. Because weights are the operand reused the most in a batched inference workload, WS minimises weight-fetch energy. This is the dataflow behind Google's TPU v1 systolic array, where a 256×256 grid is pre-loaded with the weight matrix and activations march through — see our companion piece on the systolic array explained for the mechanics.

WS shines when the weight matrix is reused across a large batch or a large feature map, which is why it suits big fully-connected and GEMM-heavy transformer layers. Its weakness: partial sums must move, and for layers with small reuse (depthwise convolutions, batch size 1) the activation and psum traffic can swamp the savings.

Output stationary (OS)

Output-stationary flips the priority: each PE owns one output element and keeps its accumulator local, reading the long chain of weight×activation products without ever spilling the partial sum to SRAM. Since the partial sum is the most write-intensive operand (it is updated on every MAC), OS slashes the energy of partial-sum movement — valuable because psum accesses often outnumber weight accesses in deep layers. ShiDianNao is the textbook OS accelerator. The trade-off is that both weights and inputs must stream in, so OS pays more to feed operands than WS does.

Row stationary (RS) and the animated intuition

Row stationary, introduced with MIT's Eyeriss chip, is the clever compromise: it does not optimise a single operand but maximises total reuse across all three. Each PE is assigned one row of the filter and one row of the input feature map; it computes a 1-D convolution, keeping the filter row, the input row, and the running partial-sum row all local. PEs are then tiled so that filter rows are reused horizontally, input rows diagonally, and partial sums accumulate vertically. The result is that Eyeriss achieved its energy efficiency primarily by exploiting this balanced local reuse rather than by brute-force clock speed.

Eyeriss reported that row-stationary cut total data-movement energy by roughly 1.4×–2.5× versus weight- or output-stationary dataflows on AlexNet and VGG layers — not by moving data faster, but by moving it far less.

Comparing the dataflows at a glance

Why this matters for your RTL and your tools

Dataflow is not an abstraction that lives only in a paper — it is baked into the RTL micro-architecture of the PE array, the NoC topology, and the address-generation logic. When you model one of these arrays in SystemVerilog and push it through Synopsys Design Compiler or Fusion Compiler for synthesis, the choice of dataflow directly shapes the register-file depth per PE, the local buffer sizing, and the on-chip interconnect bandwidth you must close timing on in Cadence Innovus. Power analysis in Synopsys PrimePower or Ansys will show the memory subsystem, not the MAC array, as the dominant consumer — exactly as the energy table predicts. Verification teams validating the accumulation ordering lean on Siemens Questa and formal flows to prove that reordered partial sums remain bit-accurate. Teams building production accelerators often bring in dedicated custom silicon design services to co-optimise the dataflow against the target PDK before committing to tape-out.

If you want to go deeper on the surrounding micro-architecture, our note on memory hierarchy for AI accelerators pairs naturally with this one, and the full treatment lives in our hardware track on the CourseTron courses page.

Ready to design your own PE array and quantify its energy per inference? Join the CourseTron AI Hardware Accelerators course — you will model WS, OS, and RS dataflows in SystemVerilog, synthesise them, and read the power reports yourself. Explore the AI Hardware track →

Frequently asked questions

Is one dataflow universally best?

No. The optimal dataflow depends on layer shape, batch size, and the memory hierarchy. Weight stationary wins for large reused weight matrices; output stationary wins when partial-sum traffic dominates; row stationary is the most robust across varied CNN layer shapes. Flexible accelerators now support reconfigurable dataflows to adapt per layer.

Why is DRAM energy so much higher than a MAC?

A DRAM access involves activating a row, driving long bitlines, crossing the chip boundary over high-capacitance I/O, and traversing a memory controller. Those physical distances and capacitances dwarf the tiny transistors doing a local multiply, giving the roughly 100–200× energy gap that dataflow optimisation targets.

How does a systolic array relate to these dataflows?

A systolic array is a hardware structure — a rhythmically pumped grid of PEs. A dataflow is the scheduling policy you run on it. A systolic array can be operated weight-stationary (like TPU v1) or output-stationary; the two concepts are orthogonal but frequently paired.

Does dataflow still matter for transformers and LLMs?

Absolutely. Attention and feed-forward blocks are large matrix multiplies, and at batch size 1 — common in LLM serving — weight reuse collapses, so weight-fetch bandwidth becomes the bottleneck. This is precisely why modern inference accelerators obsess over keeping weights resident in on-chip SRAM or high-bandwidth memory close to the compute.