All Resources
R-38
Electronics
Embedded AI: Models That Fit in Kilobytes
Running inference on a microcontroller — the memory arithmetic, quantisation-aware training, operator support, and the power budget that decides whether the product ships.
PAR2 Labs
August 21, 2026
15 min
Embedded AI is a memory and power problem wearing a machine-learning costume. The model architecture matters far less than whether the tensor arena fits in SRAM, whether every operator is supported by the runtime, and whether the duty cycle lets a coin cell last the year. Get the arithmetic right first and the model almost chooses itself.
01
The budget you are actually working inside
On a microcontroller you are allocating three scarce resources at once: flash for the model weights and code, SRAM for the tensor arena and everything else, and energy per inference. All three are fixed at design time and none can be increased later by shipping an update.
Do this arithmetic before choosing an architecture. Most projects that stall did so because the tensor arena would not fit, discovered after the model was trained.
Fig 1 — the tensor arena is peak concurrent activation memory, not total. Measuring it is the first thing to do, and the runtime will tell you exactly.
02
Quantisation, honestly
Full INT8 quantisation is what makes inference on a microcontroller possible: roughly four times smaller than float32, and integer arithmetic that a Cortex-M can actually execute quickly with CMSIS-NN kernels behind it.
It is not lossless. Post-training quantisation is fast and usually costs a little accuracy; quantisation-aware training costs a retraining cycle and usually recovers most of it. Which you need depends entirely on how much headroom the task has.
| Approach | Size vs float32 | Accuracy cost | When to use |
|---|---|---|---|
| Float32 baseline | 1× | None — this is the reference | Never ships to an MCU; it is what you measure against |
| Post-training dynamic | ~4× smaller | Small, unpredictable per class | Quick feasibility check only |
| Post-training full INT8 | ~4× smaller | Usually 1–3% top-1, concentrated in rare classes | The default. Needs a representative calibration set |
| Quantisation-aware training | ~4× smaller | Often under 1% | When PTQ loses accuracy you cannot spare |
| Pruning + INT8 | 4–10× smaller | Task dependent, needs careful evaluation | When flash is the binding constraint, not accuracy |
Always evaluate the quantised model per class, never on aggregate accuracy — quantisation loss concentrates on rare classes.
Embedded AI is a memory and power problem wearing a machine-learning costume.
03
The path from dataset to device
Eight steps, in this order. Steps 01 and 02 are the ones that decide whether the rest is possible.
01
Do the memory arithmetic on paper
TOOL
A spreadsheet
USE
Parameters × 1 byte, plus peak activations, plus buffers
GET
A go/no-go before any training
Estimate INT8 weights as one byte per parameter, add the runtime and application flash, then estimate peak concurrent activations for the candidate architecture. Compare against the datasheet, leaving at least 25% SRAM headroom.
Why: Discovering after training that the arena does not fit means changing architecture and retraining. Ten minutes of arithmetic prevents weeks of rework.
02
Capture data on the target sensor
TOOL
The actual hardware
USE
Real sensor, real placement, real noise
GET
A dataset that transfers
Collect training data from the sensor you will ship, mounted where it will be mounted, at the sample rate you will run. Include the failure conditions — loose mounting, temperature extremes, supply noise.
Why: A model trained on a different microphone or a differently-mounted accelerometer will not transfer. Domain shift at the sensor is the most common reason field accuracy collapses.
03
Choose an architecture that the runtime supports
TOOL
TFLite Micro operator list
USE
Depthwise separable convolutions, small dense heads
GET
A model that will actually convert
Check every operator against the runtime's supported list before training. Prefer depthwise separable convolutions over dense stacks, and keep the head small. Avoid exotic layers, dynamic shapes and unsupported activations.
Why: An unsupported operator is discovered at conversion time, after training. The supported list is short and reading it first is free.
04
Train, then quantise with a calibration set
TOOL
TensorFlow / LiteRT converter
USE
Full INT8, representative dataset of 100–500 samples
GET
A quantised .tflite
Train in float, then convert with full integer quantisation using a calibration set drawn from real captured data covering the full input range. Force INT8 input and output so no float ops sneak into the graph.
Why: The calibration set determines the quantisation ranges. Calibrating on unrepresentative data produces a model that clips exactly where your real signal lives.
05
Re-evaluate per class, not in aggregate
TOOL
Your held-out set
USE
Confusion matrix before and after quantisation
GET
Known, per-class accuracy
Score the quantised model against the same held-out set as the float model and compare confusion matrices class by class. Pay attention to the rare classes; that is where the loss lands.
Why: Aggregate accuracy can be unchanged while the one class that justified the product has collapsed. Only a per-class comparison surfaces that.
06
Measure the arena on device
TOOL
TFLite Micro interpreter
USE
Arena used bytes, reported at runtime
GET
The real SRAM figure
Deploy with a generously sized arena, read back the actual bytes used, then set the arena to that plus a margin. Do this on hardware, not in a simulator.
Why: Estimated arena size is routinely wrong in both directions. The interpreter will tell you the true number and it costs one build to find out.
07
Optimise kernels for the core
TOOL
CMSIS-NN
USE
Optimised INT8 kernels for Cortex-M
GET
Several times faster inference at the same accuracy
Build the runtime against CMSIS-NN so convolution and fully-connected layers use SIMD and DSP instructions where the core has them. Confirm the optimised kernels are actually linked in rather than falling back to reference kernels.
Why: Reference kernels are portable and slow. On a Cortex-M4 or M7 the optimised path is often several times faster for identical output, which converts directly into battery life.
08
Close the power budget
TOOL
A current probe
USE
Measured active current × inference time × duty cycle
GET
A defensible battery life figure
Measure real current draw during inference, in sleep, and during radio transmission. Compute energy per duty cycle and compare against the cell's usable capacity at the coldest expected temperature.
Why: Datasheet current figures are best case at room temperature. Battery chemistry loses capacity in cold, and a product specified from datasheets alone will miss its claimed life in the field.
04
The traps
These are the ones that cost weeks rather than hours, and every one is avoidable by doing an earlier step properly.
Where embedded AI projects lose time
01
Training before doing the memory arithmetic, then discovering the arena does not fit.
02
An unsupported operator found at conversion time, after the model is trained.
03
Calibrating quantisation on synthetic or unrepresentative data, so ranges clip real signal.
04
Judging the quantised model on aggregate accuracy and shipping a collapsed rare class.
05
Reference kernels silently linked instead of CMSIS-NN, leaving several times the speed on the table.
06
A power budget built from datasheet numbers rather than a measured current trace.
05
Reference
The LiteRT for Microcontrollers documentation and the CMSIS-NN reference are the two that matter most; everything else follows from them.
Primary documentation
LiteRT for Microcontrollers
The interpreter, the supported operator list, and arena sizing.
CMSIS-NN
Optimised INT8 kernels for Cortex-M, and how to confirm they are linked.
Edge Impulse
End-to-end tooling for capture, training and deployment when you want the path paved.
ONNX Runtime
Where the target is a larger MPU rather than an MCU.
Hugging Face Transformers
Starting points for small audio and vision models worth distilling down.
Key Takeaways
01
Do the flash and SRAM arithmetic before choosing an architecture, not after training.
02
Capture training data from the sensor you will ship, mounted where it will be mounted.
03
Check operator support against the runtime before you train anything.
04
Compare confusion matrices per class after quantisation — the loss lands on rare classes.
PAR2 Labs · Electronics
Work With Us