TinyML Machine Learning On Microcontrollers

In short: TinyML on a Cortex-M works when you quantize to int8, keep weights in flash, size one reusable SRAM tensor arena from the worst layer pair, run CMSIS-NN kernels that exploit SIMD MACs, and then race to sleep so the average power stays in the microwatt range.

Why Cortex-M budgets are so brutal

A typical Cortex-M4 part such as an STM32F4 or nRF52840 gives you roughly 256 KB of SRAM, 1 MB of flash and a clock between 64 and 168 MHz. A Cortex-M0+ may offer 16 KB of SRAM. There is no cache worth the name, no operating-system memory manager and usually no floating-point unit on the low end. A float32 convnet with 250,000 parameters would need 1 MB for weights alone, so it never fits. Everything in TinyML follows from three numbers: flash for the model, SRAM for the working set, and cycles per inference.

Key fact: weights are read-only and live in flash; activations are read-write and live in SRAM. Interviewers often check that you never confuse the two budgets.

Quantization: the first 4x

Post-training or quantization-aware training maps each float to an 8-bit integer using real = scale × (q − zero_point). Weights are typically symmetric per output channel (zero point 0), while activations are asymmetric per tensor. The scheme is described in the well-known integer-arithmetic-only inference paper by Jacob et al. Moving from float32 to int8 cuts weight storage by 4x, and the convolution accumulates into int32 so no precision is lost inside the dot product. A keyword-spotting depthwise-separable CNN with about 38,000 parameters therefore shrinks from roughly 152 KB to 38 KB, which fits comfortably in flash with room for firmware. For deeper detail see our primer on int8 quantization.

Memory budgeting: size the arena from the worst layer pair

TensorFlow Lite for Microcontrollers uses one statically allocated byte array, the tensor arena. A greedy planner reuses space once a tensor's last consumer has run. For a plain sequential network, the peak is roughly the maximum over layers of (input + output + scratch). Take a first convolution with a 49×10 input (490 B) and a 25×5×64 output (8,000 B): that pair already needs 8,490 B, plus an im2col scratch buffer. Later layers shrink as spatial size falls, so the first one or two layers set the arena.

Branches break this. A residual or concatenation keeps the skip tensor alive across several layers, so the peak can jump by the full skip size. This is why MobileNet-style inverted residuals with wide expansion layers are hostile to 64 KB budgets. Practical levers:

• Reduce early-layer channel counts or use a strided first layer to shrink the largest feature map.
• Prefer depthwise-separable convolutions, which cut both MACs and intermediate size.
• Fuse activation and pooling so ReLU and quantization do not need their own buffer.
• Keep the arena 16-byte aligned and add a margin; read the interpreter's reported used-bytes figure instead of guessing.

Key fact: model size (flash) and peak activation memory (SRAM) are independent. A model can have tiny weights and still overflow SRAM because one early feature map is huge.

CMSIS-NN: why the kernels are fast

Arm's open-source CMSIS-NN library supplies optimized int8 kernels such as arm_convolve_s8, arm_depthwise_conv_s8 and arm_fully_connected_s8. On Cortex-M4 and M7 with the DSP extension, the key instruction is SMLAD: a dual 16-bit multiply-accumulate that does two MACs per cycle. The kernel widens int8 to int16 pairs (using SXTB16), runs SMLAD into a 32-bit accumulator, then requantizes with a fixed-point multiplier and shift instead of any floating-point math. Convolution is reshaped via im2col into a small scratch buffer, which is why scratch size appears in your SRAM budget. On Cortex-M55 and M85, Helium (MVE) processes 128-bit vectors, giving up to 16 int8 lanes. On a Cortex-M0+ there is no SIMD, so expect several times lower throughput; the CMSIS-NN paper reports roughly 4.6x speed-up and 4.9x energy gain over naive code on M7.

Interview probe: "Why does int8 beat float on a chip with an FPU?" Because SIMD packs more MACs per cycle, memory traffic drops 4x, and the accumulator is exact.

Power and latency trade-offs

Energy per inference is E = V × I × t. At 3.3 V, 10 mA and 20 ms, that is 0.66 mJ. Running at half the clock does not halve energy: current falls less than linearly because leakage and flash power persist, and the MCU stays awake twice as long. Hence the standard answer is race to sleep: finish at the highest efficient frequency, then drop to a deep sleep state. Other levers include:

• Flash wait states: above roughly 24-30 MHz the flash needs wait states; an ART or instruction cache hides this for code but weight reads stream linearly, so benchmark real cycles.
• Duty cycling: run a cheap always-on detector (energy or zero-crossing) and wake the neural net only on a trigger.
• Latency vs accuracy: a 49×10 MFCC keyword model at 20 ms leaves slack under a 40 ms hop; a bigger model that misses the hop forces dropped frames.
• DMA: let I2S or ADC DMA fill the buffer while the core sleeps.

For silicon teams, micro-NPUs such as the Arm Ethos-U55 pair with an M-class host and offload convolution; hardware planning of that kind is covered in edge AI hardware. Verification of such accelerators uses tools like Synopsys VCS or Cadence Xcelium, and teams at Avecas design services build exactly this kind of embedded-AI SoC logic. For a quantization-versus-pruning comparison, read model compression techniques.

Key fact: battery life is dominated by sleep current and duty cycle, not peak MAC speed. A 1% duty cycle at 10 mA averages 100 uA plus sleep leakage.

What interviewers actually ask

• Compute the arena for a given layer list, including a skip connection.
• Explain per-channel versus per-tensor quantization and why zero point matters for padding.
• Describe how requantization avoids floating point.
• Argue when to choose M4 + CMSIS-NN versus M55 + Helium versus an Ethos-U NPU.

Training that connects embedded software to chip design, from RTL to firmware, is available through ChipXpert VLSI training.

Want to design and debug TinyML pipelines hands-on? Explore the AI hardware modules in our CourseTron courses.

FAQ

How much RAM does a TinyML model really need?

Enough for the peak of live activations plus scratch, usually 10 to 100 KB for audio and sensor models, independent of how large the weights are in flash.

Is int8 always the right choice?

For Cortex-M with DSP or Helium, yes in most cases. Int16 activations help when accuracy collapses, and int4 saves flash but costs unpacking cycles.

Can a Cortex-M0+ run a neural network?

Yes for tiny models, but without SIMD it is several times slower than an M4, so use very small dense or depthwise layers and aggressive duty cycling.

Why not run slower to save power?

Because static and flash current continue while the core is awake; finishing sooner and entering deep sleep usually costs less total energy.