In short: TensorFlow Lite (now branded LiteRT) uses a converter to turn a trained graph into a compact FlatBuffer with fused ops and int8 weights, a small interpreter to run it on the CPU, and pluggable delegates that hand supported sub-graphs to GPUs, DSPs and NPUs.
Why a separate runtime exists at all
A desktop TensorFlow install is hundreds of megabytes, assumes dynamic memory and floating point, and burns watts. A phone app has a few megabytes to spare; a Cortex-M4 keyboard-wake-word device has 256 KB of SRAM and a coin cell. TensorFlow Lite attacks the problem in three layers: shrink and reshape the model offline (converter), run it with a tiny deterministic engine (interpreter), and push heavy maths to silicon that is far more efficient than a CPU (delegates). If you want to see the pipeline end to end on real code first, our TensorFlow Lite worked example walks through a complete conversion.
The converter: from graph to FlatBuffer
The converter takes a SavedModel, a Keras model or a concrete function and emits a single .tflite file. Under the hood it is built on MLIR: the graph is lowered to the TFLite dialect, then optimised. The important passes are:
- Constant folding - anything computable at convert time (shape arithmetic, frozen weights) disappears from the runtime graph.
- Operator fusion - Conv2D + BatchNorm + ReLU6 collapses into one Conv2D with a fused activation; batch-norm scale and shift are folded into the filter weights and bias. One kernel launch, one memory round trip instead of three.
- Layout and op legalisation - ops are mapped to the built-in TFLite op set; anything unmapped must be allowed through Select TF Ops (large binary cost) or rewritten.
The output is a FlatBuffer: the interpreter memory-maps it and reads tensors in place with no parsing or copying, which matters when RAM is scarce.
Quantisation: the affine mapping
TFLite uses an affine scheme: real = scale × (q − zero_point). A float32 weight (4 bytes) becomes an int8 (1 byte), so the model shrinks about 4x. MobileNetV2 goes from roughly 14 MB to about 3.5 MB, and integer multiply-accumulate units are far smaller and cheaper in energy than floating-point ones, which is exactly what a silicon team wants when sizing an NPU datapath.
Key fact: TFLite int8 weights are symmetric (zero-point fixed at 0) and quantised per output channel, while activations are asymmetric and per-tensor. Symmetric weights remove a cross-term from the inner loop, so the hardware MAC stays simple.
| Mode | Needs data? | Size | Typical use |
|---|---|---|---|
| Dynamic range | No | ~4x smaller | Quick win on phone CPUs; activations stay float |
| Float16 | No | ~2x smaller | Mobile GPU delegate |
| Full integer (int8) | Yes, 100-500 samples | ~4x smaller | MCUs, DSPs, NPUs, Edge TPU |
| QAT (quantisation-aware training) | Training run | ~4x smaller | Recovering accuracy on sensitive models |
Full-integer conversion needs a representative dataset. The converter runs these samples through the float model, records min/max per tensor, and derives scale and zero-point. Poor calibration data (say, only silent audio for a keyword spotter) gives tiny ranges that clip real inputs, and accuracy collapses. Typical post-training loss on image classifiers is under 1 percent top-1; depthwise-heavy or transformer models can lose far more, which is when QAT earns its cost.
Biases are stored as int32 with scale = input_scale × weight_scale, so they add straight into the 32-bit accumulator. After accumulating, the output is requantised by a fixed-point multiplier and a right shift: no division, no float, which is why the same kernel runs on a Cortex-M0+ with no FPU.
The interpreter: small, static, predictable
The interpreter loads the FlatBuffer, resolves each op code to a kernel through an op resolver, plans tensor memory, then executes ops in topological order on Invoke(). The memory planner is the clever part: it knows each tensor's lifetime, so tensors whose lifetimes do not overlap share the same bytes in one arena. A model with 20 MB of notional activations may need only 2 MB of real working memory.
TFLite Micro takes this to the extreme for microcontrollers. There is no malloc and no OS: you hand it a static uint8_t tensor_arena[], register only the ops your model uses with MicroMutableOpResolver (dropping unused kernels saves flash), and the runtime core fits in roughly 16 KB. On Arm Cortex-M it swaps reference kernels for CMSIS-NN, which uses SIMD instructions to do two or four 8-bit MACs per cycle; vendors also ship optimised kernels for Cadence Tensilica HiFi DSPs and Synopsys ARC DSP cores.
Key fact: sizing the tensor arena is the number-one practical headache on MCUs. Too small and AllocateTensors() fails; the usual method is to start generous, read the arena-used report, then trim.
Delegates: reaching NPUs and DSPs
A delegate is a plug-in that claims part of the graph. When you call ModifyGraphWithDelegate(), the delegate reports which nodes it supports; the runtime partitions the graph into delegated sub-graphs and CPU leftovers. Each delegated sub-graph is replaced by a single custom node whose Invoke sends the tensors to the accelerator.
- GPU delegate - OpenCL/OpenGL/Metal compute shaders; great for float16 vision models.
- XNNPACK - heavily tuned CPU kernels using NEON/AVX; on by default for float on many builds.
- Hexagon / vendor NPU delegates - Qualcomm, MediaTek and Samsung expose DSP/NPU paths, historically through Android NNAPI, which is being superseded by vendor delegates in recent Android releases.
- Edge TPU - requires full int8 and a separate offline compiler; any op it cannot map runs on the host.
- Arm Ethos-U - the Vela compiler rewrites the whole int8 graph into a command stream for the NPU.
The catch is graph fragmentation. If an unsupported op sits in the middle of a network, you get two accelerator sub-graphs plus a CPU hop, and the data copies across the boundary can erase the speed-up. An interview favourite: "Why did enabling the NPU delegate make inference slower?" The answer is usually fallback ops, fragmentation and fp32-to-int8 boundary conversions.
What this means for silicon and flow engineers
Quantised graphs define the hardware spec: int8 MAC arrays, per-channel scale registers, a requantise stage with a saturating shifter, and SRAM sized to the tensor arena. Teams commonly model such blocks in C++ and run high-level synthesis, for example with Siemens Catapult, then verify RTL in simulators from Synopsys or Cadence before tape-out. If you want to work on such NPUs professionally, the engineers at Avecas build exactly this kind of design and verification flow; for hands-on training see ChipXpert. For a wider view of accelerator choices, read our edge AI accelerators compared guide.
Want to build and debug this pipeline yourself? Explore the AI hardware modules in our CourseTron courses and start converting real models today.
FAQ
Is TensorFlow Lite the same as LiteRT?
Yes. Google renamed TensorFlow Lite to LiteRT in 2024 and made it framework-neutral, so it also ingests PyTorch and JAX models. The .tflite format and the concepts here still apply.
Why does my int8 model lose accuracy?
Usually poor calibration data, activation outliers that stretch the range, or layers like depthwise convolutions with wide per-channel spreads. Use a better representative dataset, keep sensitive layers in float, or use quantisation-aware training.
Can TFLite run without an operating system?
Yes. TFLite Micro needs no OS, no dynamic allocation and no C++ standard library beyond basics; you supply a static tensor arena and an op resolver.
Why is my delegate slower than the CPU?
Unsupported ops split the graph into several partitions, and each hand-off copies data and may convert types. Check which nodes were delegated and fix or replace the offenders.