TPU (Tensor Processing Unit) architecture

In short: A TPU accelerates deep learning by streaming int8 activations and weights through a large systolic array of multiply-accumulate cells, keeping data on-chip in local buffers so that thousands of MACs stay busy every cycle without hammering external DRAM.

Google's Tensor Processing Unit (TPU) is the canonical example of a domain-specific architecture built for one job: the dense matrix multiplies and convolutions at the heart of neural networks. Where a CPU spends most of its transistors on caches, branch prediction and out-of-order logic, a TPU spends them on raw arithmetic. The first-generation TPU shipped a 256×256 array of 8-bit multiply-accumulate (MAC) units -- 65,536 of them -- delivering roughly 92 tera-operations per second while drawing on the order of 40 W of core power. Understanding why that design wins is one of the most interview-relevant topics in AI hardware today.

Why a Systolic Array?

The bottleneck in neural network inference is not arithmetic throughput in isolation -- it is feeding the arithmetic units. A naive design reads two operands and writes one result to a register file for every multiply, and that register-file and memory traffic dominates energy. A systolic array sidesteps this. Coined by H.T. Kung and Charles Leiserson in 1978, the idea is to arrange processing elements (PEs) in a grid and let data rhythmically "pump" through them -- like blood through a heart, hence systolic. Each operand is read from memory once, then reused by every PE it passes.

In the TPU's weight-stationary scheme, a tile of the weight matrix is loaded into the PEs and held there. Activations then stream in from the left, one column per clock. At every PE, the incoming activation is multiplied by the resident weight, added to the partial sum arriving from above, and the running sum is passed downward. After the pipeline fills, one full column of the output matrix emerges every single cycle. For a 256×256 array that is 65,536 MACs per clock with each weight fetched exactly once.

Key fact: Reading a 32-bit word from off-chip DRAM costs on the order of 640 pJ, while a single 8-bit MAC costs well under 1 pJ. The systolic array wins not by multiplying faster but by multiplying without moving data.

Int8 Quantization: Doing More With Fewer Bits

TPUs lean hard on int8 quantization. A trained network's weights and activations, normally float32, are mapped to 8-bit integers using an affine scheme: real = scale × (q − zero_point). An 8-bit multiplier is roughly 6× smaller in area and energy than a float32 multiplier, so you can pack far more of them into the same silicon budget, and you move a quarter of the bytes.

The subtle part is accumulation. Multiplying two int8 values can produce a 16-bit product, and summing 256 of them needs headroom, so the TPU accumulates into 32-bit registers before requantizing the result back to int8 for the next layer. Done carefully -- with per-channel scales and a calibration pass over representative data -- int8 inference typically holds accuracy within a fraction of a percent of the float baseline for CNNs and many transformers.

On-Chip Memory and the Roofline

None of this matters if the array starves. The TPU surrounds the matrix unit with a large on-chip SRAM -- the first-generation "Unified Buffer" was 24 MiB -- that stages activations between layers, plus a dedicated weight FIFO. Keeping intermediate feature maps resident means a convolution's output feeds directly into the next layer without a DRAM round-trip. The chip becomes compute-bound rather than memory-bound, the favorable side of the classic roofline model.

The animation below shows a single activation byte entering the array and being reused across successive PEs on each tick -- the essence of why one DRAM read serves many multiplies.

Key fact: A systolic array's throughput ramps up over a fill latency of roughly 2N cycles for an N×N array, so TPUs favor large batch sizes and big matrices that keep the pipeline saturated -- a key difference from latency-optimized GPUs.

Building One in RTL

A systolic MAC array is a favorite teaching design precisely because it is so regular: one PE cell replicated across a parameterized grid. In practice you would describe the PE in SystemVerilog, synthesize with Synopsys Design Compiler or Cadence Genus, and run place-and-route in Cadence Innovus, where the array's uniform structure gives clean, congestion-free floorplans. Timing closure centers on the MAC's accumulate path, and functional sign-off leans on Siemens Questa for simulation. This is exactly the kind of accelerator block our learners build end-to-end; if your team needs production silicon rather than coursework, Avecas offers full RTL-to-GDSII design services.

Where TPUs Fit

TPUs excel at high-throughput, batched inference and large-scale training of dense networks; they are less suited to irregular, branch-heavy workloads where a CPU or a flexible GPU shines. For a side-by-side on programmability, memory models and ecosystem, see our deep dive on GPU vs TPU for AI, and for the number format trade-offs that drive accelerator design read int8 quantization explained.

Ready to design your own matrix engine? Join the CourseTron AI Hardware & Accelerator Design track and build a synthesizable systolic array from RTL to layout.

Frequently Asked Questions

What is the difference between a TPU and a GPU?

A GPU is a massively parallel SIMT processor with thousands of general-purpose cores and a rich software stack; a TPU is a narrower domain-specific chip whose systolic matrix unit is purpose-built for dense multiply-accumulate. TPUs typically win on throughput-per-watt for large batched matrix workloads, while GPUs offer more flexibility for varied kernels.

Why does the TPU use int8 instead of float32?

Int8 multipliers are far smaller and cheaper in energy than float32, letting the chip pack tens of thousands of MAC units and move a quarter of the data. With calibration and 32-bit accumulation, inference accuracy stays within a fraction of a percent of the float baseline for most networks.

What is a systolic array?

It is a grid of simple processing elements through which operands flow rhythmically, so each value read from memory is reused by many compute cells. This maximizes arithmetic per byte fetched, which is the dominant efficiency lever in neural network hardware.

Can I prototype a TPU-style array on an FPGA?

Yes. A parameterized systolic MAC array maps naturally onto FPGA DSP slices and is a common capstone project. You trade clock speed and density versus an ASIC, but the dataflow and quantization logic are identical.