In short: An NPU is a specialized matrix-multiply engine that runs neural-network inference on-device at a fraction of the energy a CPU or GPU would burn, which is why your phone can transcribe speech, blur backgrounds, and run a chatbot without touching the cloud.
Open the spec sheet of any 2024-era flagship phone and you will find a number quoted in "TOPS" next to the CPU and GPU: 35 TOPS on an Apple A17 Pro Neural Engine, roughly 45 TOPS on a Qualcomm Snapdragon 8 Gen 3 Hexagon, and 40+ TOPS on the NPUs shipping in Copilot+ laptops. That block is the Neural Processing Unit, and it has quietly become the third major compute engine in modern silicon. This article explains what an NPU actually is at the hardware level, why it exists alongside the CPU and GPU, and how engineers map a trained model onto one.
Why a dedicated engine at all?
Neural-network inference is dominated by one operation: the multiply-accumulate (MAC). A convolution layer or a transformer's attention projection is, underneath, billions of acc += a * b steps over large tensors. A general-purpose CPU can do this, but it spends most of its transistor budget and energy on things inference does not need: branch prediction, out-of-order scheduling, large caches, and precise per-instruction control. A GPU is far better because it is already a wide SIMD machine, but it was architected for graphics and carries fixed-function units and a memory hierarchy tuned for that workload.
The NPU throws away everything that inference does not use and spends the entire die area on MAC throughput and on keeping that MAC array fed. The result is a dramatic improvement in the metric that matters on a battery: energy per inference, usually quoted as TOPS/Watt. A CPU might deliver well under 1 TOPS/W on INT8 matrix work; a modern mobile NPU reaches 10-15 TOPS/W. That order-of-magnitude gap is the entire reason the block exists.
Key fact: "TOPS" counts trillions of INT8 operations per second, where one MAC is conventionally two operations (a multiply and an add). A 40-TOPS NPU is doing on the order of 20 trillion multiply-accumulates every second.
The core: a systolic MAC array
Most NPUs are built around a systolic array — a 2D grid of tiny processing elements (PEs), each of which holds a weight, multiplies it by an incoming activation, adds the running partial sum, and passes data to its neighbour on the next clock. Google's TPU popularised a 256×256 array (65,536 MACs). Mobile NPUs use smaller tiles, often 32×32 or several 16×16 blocks, that are easier to keep busy on irregular layer shapes.
The subtlety is not the arithmetic — multipliers are cheap — it is the memory. Fetching an INT8 operand from external LPDDR costs roughly 100-1000× the energy of the multiply itself. So NPUs are really dataflow machines designed to move each weight and activation from DRAM to on-chip SRAM once, then reuse it across as many MACs as possible. This is the principle behind "weight-stationary" and "output-stationary" dataflows, and it is why on-chip SRAM capacity, not peak TOPS, is often the real bottleneck for large models.
Animated: how one weight gets reused
Quantization: why NPUs love INT8
Models are trained in 32-bit or 16-bit float, but running inference in float wastes NPU silicon. NPUs are built for low-precision integer math — INT8 is the workhorse, with INT4 and even INT2 now common for LLM weights. An INT8 multiplier is roughly a quarter the area and far lower energy than an FP32 one, and you can pack four INT8 values into the datapath that would hold one FP32.
Getting there requires quantization: mapping float ranges to integer scales, typically q = round(x / scale) + zero_point. Post-training quantization (PTQ) is one calibration pass; quantization-aware training (QAT) simulates the rounding during training to recover accuracy. A well-quantized INT8 vision model usually loses under 1% top-1 accuracy — a trade most edge applications accept gladly for the 4× energy win.
Key fact: Transformer LLMs on-device increasingly use INT4 weights with INT8 or FP16 activations ("W4A8/W4A16"), because the weights dominate memory and are more tolerant of coarse quantization than the activations.
Where the NPU fits in SoC design
The NPU is an IP block integrated alongside CPU, GPU, ISP, and DSP on a single SoC, sharing the system memory bus and often a slice of system-level cache. Designing or integrating that block is a full physical-design job: the systolic array is a dense, regularly-structured datapath that demands careful floorplanning, clock-tree synthesis, and power-grid planning. Teams drive this with the standard EDA stack — synthesis in Synopsys Design Compiler or Cadence Genus, place-and-route in Cadence Innovus or Synopsys IC Compiler II, and signoff timing/power with Synopsys PrimeTime. Functional and power verification of the accelerator's control logic typically runs on Synopsys VCS or Siemens Questa, with formal and low-power (UPF) checks layered on top.
| Engine | Best at | Typical INT8 efficiency |
|---|---|---|
| CPU | Control, branching, small/irregular tensors | <1 TOPS/W |
| GPU | Large batched parallel math, training | 2-5 TOPS/W |
| NPU | Fixed-topology inference at low power | 10-15 TOPS/W |
The software path: from model to NPU
Hardware is only half the story. A trained model (say a PyTorch export) is converted to an intermediate format — ONNX, TensorFlow Lite, or Core ML — then handed to a vendor runtime: Qualcomm's QNN/AI Engine, Apple's Core ML + Neural Engine, Arm's , or the cross-vendor path via ONNX Runtime. A compiler quantizes the graph, fuses operators (e.g. conv+bias+ReLU into one pass), and lowers everything to the NPU's instruction set. Operators the NPU cannot handle "fall back" to the CPU or GPU — and those fallbacks, with their round-trips through shared memory, are frequently the real performance cliff engineers spend their time eliminating.
If you want to go deeper on the broader landscape of accelerators beyond the phone, see our companion piece on edge AI hardware, and for the arithmetic itself our explainer on INT8 quantization for inference. For hands-on RTL and physical-design practice on accelerator blocks, our team also runs industry training through ChipXpert's VLSI programs.
Want to design accelerator datapaths, not just read about them? The CourseTron AI-Hardware track walks you from MAC-array microarchitecture to an INT8 inference block through full synthesis and place-and-route.
FAQ
Is an NPU the same as a GPU?
No. A GPU is a wide, programmable parallel processor originally built for graphics and excellent for training and batched math. An NPU is a narrower fixed-function inference engine that sacrifices flexibility for far higher energy efficiency on INT8/INT4 neural workloads at low power.
What does "TOPS" actually measure?
Trillions of operations per second, almost always at INT8 precision. One multiply-accumulate counts as two operations. On its own, TOPS is a peak number; real throughput depends on memory bandwidth, SRAM size, and how well the model maps to the array.
Does running AI on an NPU hurt model accuracy?
Slightly, because the model is quantized to integer precision. With quantization-aware training, INT8 vision models typically stay within about 1% of their float accuracy, and modern INT4 LLM weight schemes keep quality acceptable for on-device use.
Why not just do everything in the cloud?
On-device inference wins on latency (no network round-trip), privacy (data never leaves the phone), offline availability, and cost. The NPU is what makes that economically and thermally feasible on a battery.