In short: Client SoCs add an NPU because a fixed-function matrix engine delivers several times more inferences per watt than a CPU or GPU, which lets AI features run always-on and on battery, but its real-world speed is capped by the shared system memory bandwidth, not by its headline TOPS number.
Why a phone or laptop SoC carries an NPU at all
A modern phone or thin laptop has a power envelope of roughly 3 to 15 watts sustained, and that budget is shared by the display, radios, CPU, GPU and memory. AI features such as background blur, speech-to-text, photo enhancement, live captions and wake-word detection are not occasional jobs. They run continuously, often for hours. Running them on the CPU wastes energy on instruction fetch, branch prediction and out-of-order machinery that a neural network never needs. Running them on the GPU is better, but the GPU is built for wide floating-point shader work and wakes a large power domain.
A neural processing unit (NPU) removes the overhead. Its datapath is a dense array of low-precision multiply-accumulate (MAC) units, fed from local SRAM through a hard-wired dataflow. Because the operation is known in advance (convolution, matrix multiply, activation), the control logic is tiny. The result is typically 5x to 20x better energy per operation than a CPU on INT8 work. For a deeper look at the architectural differences, read our comparison in NPU vs GPU.
Always-on inference: the real reason
The most important NPU use case is not a flashy demo. It is the tiny model that never sleeps: a keyword spotter listening for a wake phrase, a presence-detection model watching a low-resolution camera so the laptop locks when you leave, or a context classifier deciding whether the phone is in a pocket. These models are small (tens of kilobytes to a few megabytes) and run at a handful of milliwatts or less.
To achieve this, SoC architects place a micro-NPU or a DSP-with-MAC array in an always-on power domain, with its own small SRAM so that DRAM can stay in self-refresh. Only when a confidence threshold is crossed does the main NPU, and then the CPU, power up. The saving comes from power gating, not from clock speed.
Key fact: In an always-on design, standby energy is dominated by SRAM leakage and DRAM refresh, not by the MAC array. That is why a 1 mW micro-NPU uses its own retained SRAM instead of DRAM.
What a TOPS rating really means
TOPS means tera operations per second, and one MAC counts as two operations (a multiply and an add). The peak figure is simple arithmetic: TOPS = 2 x number of MACs x clock frequency. An NPU with 16,384 INT8 MACs at 1.4 GHz gives 2 x 16,384 x 1.4e9, about 45.9 TOPS. This is the origin of the 40+ TOPS requirement Microsoft set for its Copilot+ PC class.
Treat the number with caution for three reasons:
- Precision matters. Vendors quote INT8 or even INT4 peak. A 45 TOPS INT8 NPU may deliver half that for FP16. Always ask which data type the number refers to.
- Sparsity inflation. Some figures assume structured 2:4 sparsity, which doubles the headline but only helps models pruned for it.
- Utilisation. A depthwise convolution or a small-batch matrix-vector multiply may use under 10 percent of the array because it lacks data reuse.
An honest comparison uses TOPS per watt on a named model (for example MobileNet or a 7B parameter LLM), together with measured latency.
The memory wall: shared bandwidth sets the ceiling
Large language models expose the weakness of TOPS. During token generation, the NPU reads every weight once per token and does only about two operations per weight. That is a memory-bound workload. A 7B model quantised to INT4 occupies roughly 3.7 GB. A laptop with a 128-bit LPDDR5X-8533 interface provides 8533 MT/s x 16 bytes, about 136 GB/s, at best. Dividing gives an upper bound near 36 tokens per second, even if the NPU had unlimited compute. In practice the CPU, GPU, display and camera pipeline are also pulling from the same pool, so sustained figures are lower.
Key fact: Halving weight precision roughly doubles decode speed on a bandwidth-bound NPU, which is why INT4 and grouped quantisation are standard on client devices.
Designers fight the wall in several ways: a large system-level cache so activations stay on chip, tiling that keeps weights resident in NPU SRAM, compressed weights decompressed at the array edge, and wider or faster DRAM. Apple's unified-memory approach and on-package LPDDR attack the same limit from the packaging side. The memory bandwidth for AI accelerators article covers HBM and GDDR options, which phones cannot afford.
How NPU, GPU and CPU split the work
The operating system's ML runtime (Core ML, Windows ML with ONNX Runtime, Android NNAPI or vendor delegates) partitions the model graph and assigns each segment to an engine. The rules of thumb are:
| Engine | Best at | Typical client tasks |
|---|---|---|
| NPU | Sustained INT8/INT4 tensors at low power | Blur, noise suppression, captions, small LLMs, wake words |
| GPU | High-throughput FP16, flexible shaders | Image generation, large or novel models, burst performance |
| CPU | Control flow, unsupported ops, small tensors | Tokenisation, pre/post-processing, fallback layers |
Fallback is a hidden cost. If one unsupported operator forces a graph split, tensors cross the fabric to the CPU and back, destroying the energy advantage. Good NPU enablement is therefore mostly a compiler and operator-coverage problem.
Designing and verifying an NPU in practice
NPU teams synthesise the MAC array with tools such as Synopsys Fusion Compiler or Cadence Genus and Innovus, because a regular array is highly sensitive to placement and clock-tree quality. Power intent is written in UPF so that the always-on domain, the NPU domain and the retention SRAM can be isolated and gated, and verified with static checkers. Functional verification uses Siemens Questa or Cadence Xcelium with randomised convolution layers checked against a software golden model, and emulation for full-network runs. If you want to build these skills hands on, our CourseTron courses cover the flow, and teams that need silicon help can look at Avecas design services.
Related reading: NPU vs GPU and memory bandwidth for AI accelerators.
Want to design low-power AI silicon yourself? Explore the AI hardware and physical design tracks in the CourseTron course catalogue and for instructor-led VLSI training see ChipXpert.
FAQ
Is a higher TOPS number always a faster NPU?
No. TOPS is a peak compute figure at a specific precision. Real speed depends on utilisation, operator support and memory bandwidth, and for LLMs bandwidth is usually the limiter.
Why not just run everything on the GPU?
The GPU can, and it often does for bursty or large jobs, but it draws far more power per inference. For continuous tasks on battery, the NPU wins on energy and thermals.
Can an NPU run a large language model?
Yes, small quantised models of about 1B to 8B parameters, provided the weights fit in RAM and the bandwidth supports a usable token rate.
What is the difference between the NPU and a DSP?
A DSP is a programmable vector processor; an NPU is a mostly fixed-function tensor engine. Many always-on blocks combine both.