TensorFlow Lite Example

? Print / Save as PDF

In short: Converting a small Keras CNN to TensorFlow Lite takes five lines, and adding full-integer post-training quantisation with a few hundred calibration samples shrinks the model about 4x and speeds up CPU inference by roughly 2x, for a typical accuracy cost under one percentage point.

If you have read our overview of TensorFlow Lite, you know the theory: a flatbuffer model, a tiny interpreter, optional delegates. This article is the hands-on companion. We take a small image classifier, convert it, quantise it, run it through the interpreter and look at the numbers. Every step maps to a question interviewers for edge-AI roles like to ask.

The model we are shrinking

We use a deliberately small network for 28x28 grayscale images (Fashion-MNIST works well): Conv(16, 3x3) → MaxPool → Conv(32, 3x3) → MaxPool → Flatten → Dense(64) → Dense(10). Count the parameters: 160 + 4,640 + 51,264 + 650, about 56,700 weights. At 4 bytes each that is roughly 227 KB of weights. The Dense layer after Flatten holds 90% of them, which matters later because it dominates the size saving.

import tensorflow as tf
model = tf.keras.Sequential([
    tf.keras.layers.Input((28, 28, 1)),
    tf.keras.layers.Conv2D(16, 3, activation="relu"),
    tf.keras.layers.MaxPool2D(),
    tf.keras.layers.Conv2D(32, 3, activation="relu"),
    tf.keras.layers.MaxPool2D(),
    tf.keras.layers.Flatten(),
    tf.keras.layers.Dense(64, activation="relu"),
    tf.keras.layers.Dense(10)])
model.compile("adam", tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True),
              metrics=["accuracy"])
model.fit(x_train, y_train, epochs=5)   # ~90% test accuracy

Step 1: plain float conversion

conv = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_fp32 = conv.convert()
open("fmnist_fp32.tflite", "wb").write(tflite_fp32)

Behind the call, the converter lowers the Keras graph to TFLite ops, fuses ReLU into the preceding Conv2D and Dense kernels, and strips training-only nodes. The file lands at about 230 KB, essentially the weight bytes plus a little metadata. Notice that nothing was lost: float conversion is numerically near-identical to Keras.

Key fact: Operator fusion is a free win. Fusing activation into a convolution removes a full read and write of the feature map, which on a memory-bound microcontroller often matters more than the arithmetic.

Step 2: post-training quantisation

Quantisation maps each float to an 8-bit integer using an affine rule: real = scale × (q − zero_point). Weights are quantised per output channel (symmetric, zero point 0); activations are quantised per tensor (asymmetric). Because activation ranges depend on real data, we give the converter a representative dataset so it can observe the min and max of every tensor.

def rep_data():
    for img in x_train[:300]:
        yield [img[None, ...].astype("float32")]

conv = tf.lite.TFLiteConverter.from_keras_model(model)
conv.optimizations = [tf.lite.Optimize.DEFAULT]
conv.representative_dataset = rep_data
conv.target_spec.supported_ops = [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]
conv.inference_input_type = tf.int8
conv.inference_output_type = tf.int8
tflite_int8 = conv.convert()

Setting the supported ops to the INT8 set forces a failure if any op cannot be quantised, which is what you want: silent float fallback is how people ship "int8" models that still run float kernels. Input and output types are set to int8 so no float conversion nodes remain, which is what an NPU or a microcontroller target such as TFLite Micro requires.

Step 3: running the interpreter

The interpreter allocates one arena for all tensors, then you set input, invoke and read output. For an int8 model you must quantise your own input using the parameters stored in the model; this is the most common bug in first attempts.

interp = tf.lite.Interpreter(model_content=tflite_int8)
interp.allocate_tensors()
inp = interp.get_input_details()[0]
out = interp.get_output_details()[0]
scale, zp = inp["quantization"]

def predict(img):
    q = np.round(img / scale + zp).astype(np.int8)
    interp.set_tensor(inp["index"], q[None, ..., None])
    interp.invoke()
    return interp.get_tensor(out["index"]).argmax()

Skip the round and the cast and you feed truncated values; accuracy collapses to near chance and the model looks "broken" when only the input handling is wrong.

Measured results

The figures below are from a representative run on a laptop-class x86 CPU, single thread, 10,000 test images, latency as a median of repeated Invoke calls. Treat them as typical rather than universal; your CPU, XNNPACK build and thread count will move the latency column.

Key fact: The 4x weight reduction is an upper bound. Biases stay int32 and every channel carries a scale, so small models land nearer 3x; large models approach 4x.

Why the speed-up appears (and when it does not)

Int8 kernels move a quarter of the bytes and use SIMD instructions that process four times as many lanes per register (NEON dot-product on Arm, VNNI on recent Intel). On a CPU, speed-up is typically 1.5x to 3x. On a microcontroller without an FPU, such as a Cortex-M0+, it is far larger because float is emulated in software. Conversely, a GPU delegate often prefers float16, and an op that lacks an int8 kernel falls back and can erase the gain; profile with the benchmark tool before promising anything.

The same arithmetic drives silicon design: an 8-bit multiply-accumulate costs a small fraction of the energy and area of a 32-bit float MAC, which is why NPUs are int8-first. Teams at companies such as Avecas that build accelerator RTL size their MAC arrays around exactly these quantised formats, then verify them with Synopsys VCS or Cadence Xcelium testbenches against a TFLite golden model. For more on that hardware angle, see int8 MAC arrays for edge AI.

Interview traps worth knowing

• Calibration data must look like production data. Calibrating on 300 black images yields tiny activation ranges and heavy clipping.
• Check per-class accuracy, not just the average. A 0.3-point drop overall can hide a 5-point drop on a rare class.
• Quantisation-aware training is the fallback when post-training loses more than about 1-2 points, typically on small or depthwise-heavy networks like MobileNet variants.
• Verify numerically: compare float and int8 logits on 1,000 images and look at the maximum deviation before trusting accuracy alone.

Want to go from here to deploying on a board and understanding the hardware beneath? Our AI hardware track at ChipXpert covers the silicon side, and the CourseTron course catalogue covers the software side.

Ready to build edge-AI skills? Explore the AI Hardware courses on CourseTron and learn quantisation, accelerators and deployment with guided labs: browse courses.

Frequently asked questions

Does post-training quantisation need retraining?

No. It only needs a few hundred unlabelled representative samples to calibrate activation ranges. Retraining enters only with quantisation-aware training, used when accuracy loss is unacceptable.

Why is my int8 model not faster?

Usually an operator lacks an int8 kernel on your target, so the runtime falls back to float and inserts quantise and dequantise nodes. Check the model in Netron and run the TFLite benchmark tool to find them.

What is the difference between dynamic-range and full-integer quantisation?

Dynamic-range quantises only weights and computes activations in float, needing no calibration data. Full-integer quantises weights and activations and needs a representative dataset, but unlocks integer-only hardware.

Can I run the .tflite file on a microcontroller?

Yes, via TensorFlow Lite for Microcontrollers, which embeds the flatbuffer as a C array. Use a full-integer model, as most MCU kernels are int8-only.