NPU vs GPU

In short: An NPU wins on TOPS/W because it hard-wires the one thing neural networks do constantly (INT8 multiply-accumulate with local data reuse) and removes the instruction overhead a GPU pays for flexibility, but you pay for that efficiency with a narrower operator set and a far less mature software stack.

Why this comparison matters

Every phone SoC, automotive chip and edge camera now ships a neural processing unit (NPU), yet data centres still run on GPUs. Interviewers love this topic because the answer is not "NPUs are faster". It is a question about where energy goes, how much of the workload you can predict at tape-out, and who writes the compiler. If you have already read our comparison of GPU vs TPU, think of the NPU as the same idea pushed into a milliwatt-to-watt power budget.

Where the energy actually goes

Mark Horowitz's ISSCC 2014 numbers for a 45 nm process are the classic starting point. An 8-bit integer add costs roughly 0.03 pJ and an 8-bit multiply about 0.2 pJ. Reading a 32-bit word from a small on-chip SRAM costs around 5 pJ, and a 32-bit DRAM access around 640 pJ. The arithmetic is almost free; moving the operand is what burns power. Process scaling shrinks all of these, but the ratio between compute and DRAM access stays brutal.

Key fact: One DRAM read costs about as much energy as thousands of INT8 multiplies. Any accelerator architecture is really a strategy for avoiding DRAM trips.

That single observation explains both designs. A GPU hides memory cost with huge thread counts, large register files and caches. An NPU removes it structurally by letting each fetched operand feed many MACs before it is evicted.

Fixed MAC arrays and dataflow

The Google TPU v1 is the textbook case: a 256 x 256 systolic array of 65,536 INT8 MACs clocked at 700 MHz in 28 nm. Each MAC is two operations, so 65,536 x 2 x 0.7 GHz is about 92 TOPS. Weights are preloaded and held still (weight-stationary), activations stream through, and partial sums ripple down the columns. No MAC ever fetches an instruction or reads a register file; its operands arrive from its neighbour on the previous clock edge.

Watching the data move

The animation below shows one activation value entering a row of the array and being multiplied by stationary weights at each stop, while a partial sum travels downward. The point to notice: after the initial load, nothing goes back to memory until the column is finished.

SIMT flexibility: what the GPU pays for

A GPU streaming multiprocessor schedules warps of 32 threads, each with its own registers and program counter state. That lets you run a convolution, a custom attention kernel, a sort and a ray-tracing pass on the same silicon. The cost is overhead: instruction fetch and decode, scoreboard logic, register-file banking, and caches that must handle arbitrary access patterns. NVIDIA's tensor cores recover much of the efficiency by being small fixed matrix engines inside the SM, which is why modern GPUs are really hybrids. Even so, an H100 delivers roughly 2 POPS of dense INT8 at up to 700 W, around 2.8 TOPS/W, while dedicated edge NPUs commonly advertise 5 to 20 TOPS/W at the chip level.

Key fact: TOPS/W is only meaningful with the precision, sparsity assumption, and utilization stated. Peak TOPS counts 2 ops per MAC and often assumes 2:4 structured sparsity.

Why INT8 is the sweet spot

An 8-bit multiplier needs roughly one-tenth to one-twentieth of the energy and area of an FP32 unit. Post-training quantization of CNNs and many transformers to INT8 loses well under 1 percent accuracy when calibrated properly, and quantization-aware training closes more of the gap. The MAC accumulates into 24 or 32 bits to avoid overflow, then a requantization stage rescales the result back to 8 bits. LLMs push the frontier further to INT4 and FP8 weights, which both architectures now support; NPUs usually add them via a reconfigurable multiplier rather than a new array.

Head-to-head comparison

The software stack is the real moat

The most common failure mode in NPU projects is not silicon; it is operator fallback. A model contains one unsupported layer, the runtime partitions the graph, and the data shuttles between NPU and CPU on every inference, destroying the efficiency you bought. NPU vendors ship graph compilers (for example Qualcomm's QNN, Arm Ethos-U via Vela, or open stacks such as TVM and ONNX Runtime execution providers) that fuse layers, tile tensors to fit on-chip SRAM, and schedule DMA double-buffering. A GPU, by contrast, will run a new attention variant the day the paper appears.

Key fact: On an NPU, SRAM tile size chosen by the compiler often decides performance more than the MAC count does.

From a design-flow view, NPU arrays are regular, which makes them friendly to physical design: replicated MAC tiles, abutted placement, and predictable clock trees in tools such as Cadence Innovus or Synopsys Fusion Compiler. Power sign-off with Synopsys PrimePower or Cadence Joules matters because switching activity from real INT8 workloads drives the TOPS/W you can actually claim. If you want to work on such blocks professionally, Avecas design services does RTL-to-GDS work on accelerator IP.

How to choose

• Fixed, well-understood vision or speech models on battery power: NPU.
• Rapidly changing architectures, training, or mixed workloads: GPU.
• Large-scale inference with stable model families: consider a data-centre ASIC, as discussed in memory bandwidth in AI accelerators.
• Hybrid SoCs: NPU for the always-on path, GPU for everything the NPU cannot map.

Learn the physical-design side of these chips through our CourseTron courses, or pair it with hands-on VLSI training at ChipXpert.

Ready to design AI accelerators yourself? Explore the AI hardware and VLSI tracks in the CourseTron course catalogue.

FAQ

Is an NPU always more power efficient than a GPU?

Only on the operators it implements natively and at high utilization. If layers fall back to the CPU, or the array sits idle waiting on memory, the advantage shrinks or vanishes.

What does TOPS mean and why is it misleading?

TOPS is tera-operations per second, counting each MAC as two operations. It is a peak figure that ignores memory stalls, precision and sparsity assumptions, so compare measured inferences per second per watt on your own model.

Why do NPUs favour INT8?

INT8 multipliers are far smaller and cheaper in energy than floating point, and most inference networks tolerate 8-bit quantization with negligible accuracy loss.

Can a GPU replace an NPU in a phone?

Technically yes, but the thermal and battery budget makes it impractical for always-on tasks such as wake-word detection or camera enhancement, which is why SoCs carry both.