A hands-on demo:
- take a YOLO v26 segmentation model
- shrink it
- measure what breaks
- run it live on a phone
┌──────────────────────────────────────────────────────────────────┐
│ Cloud / Desktop │
│ GPU · 24 GB RAM · unlimited power │
└──────────────────────────┬───────────────────────────────────────┘
│ big, slow, polluter
▼
┌──────────────────────────────────────────────────────────────────┐
│ Embedded / Mobile / Browser │
│ limited RAM · battery · no GPU │
└──────────────────────────┬───────────────────────────────────────┘
│ small, fast, frugal
▼
| Goal | Metric | Edge constraint |
|---|---|---|
| Smaller download | File size (MB) | network |
| Lower latency | Inference time (ms) | Real-time at 30 fps |
| Less memory | Peak RAM (MB) | 2–4 GB shared with OS |
| Less energy | Joules per frame? | Battery life |
☑ Measure file size (did it shrink?)
☑ Measure mAP50 / mAP50-95 (did accuracy drop?)
☑ Measure FPS on video (did it speed up?)
☑ Result check (are predictions still correct?)
The tradeoff triangle — pick two:
Accuracy
▲
╱ ╲
╱ ╲ Quantization =
╱ ╲ sacrifice accuracy
╱ ╲
▼─────────▼
Size ◄──────► Latency
Two recommended builds for any embedding project:
| Build | Format |
|---|---|
| Quality | FP16/FP32 |
| Fast | INT8 |
┌───────────────────────────────────────────────────────────────────┐
│ Step 1 Export │
│ Download the model │
└────────────────────────────────┬──────────────────────────────────┘
│
▼
┌───────────────────────────────────────────────────────────────────┐
│ Step 2 Calibrate │
│ Run FP32 model on 80 sample frames → diagnostic report │
└────────────────────────────────┬──────────────────────────────────┘
│
▼
┌───────────────────────────────────────────────────────────────────┐
│ Step 3 Convert │
│ Under the hood: .pt → ONNX → SavedModel → onnx2tf → .tflite │
└────────────────────────────────┬──────────────────────────────────┘
│
▼
┌───────────────────────────────────────────────────────────────────┐
│ Step 5 Evaluate │
│ Measure: mAP50, mAP50-95, precision, recall │
│ Analyze confusion matrices │
└────────────────────────────────┬──────────────────────────────────┘
│
▼
┌───────────────────────────────────────────────────────────────────┐
│ Step 6 Bench │
│ For each example × each model variant: │
│ Measure: FPS, avg inference time, avg confidence │
└────────────────────────────────┬──────────────────────────────────┘
│
▼
┌───────────────────────────────────────────────────────────────────┐
│ Step 7 Summarize │
│ sources, artifacts, evaluation, (FPS), │
│ benchmarks, calibration, GPU info │
└───────────────────────────────────────────────────────────────────┘
Detection (boxes) vs Segmentation (pixel masks):
Detection Segmentation
┌────────────────────────┐ ┌────────────────────────┐
│ │ │ │
│ ┌──────────┐ │ │ ░░░░░░░░░░ │
│ │ person │ │ │ ░ person ░ │
│ │ 87% │ │ │ ░ 87% ░ │
│ └──────────┘ │ │ ░░░░░░░░░░ │
│ │ │ │
└────────────────────────┘ └────────────────────────┘
Faster, smaller model More visual, ~2× heavier
Segmentation models:
| Model | Params | PT size | ONNX FP32 | Best for |
|---|---|---|---|---|
| yolo26n-seg | 2.7 M | 6 MB | 11 MB | demonstrator |
| yolo26s-seg | 9.7 M | 20 MB | 39 MB | best int8 accuracy |
| yolo26m-seg | 21.2 M | 44 MB | 85 MB | production accuracy |
| yolo26l-seg | 47.0 M | 97 MB | 187 MB | large objects, high accuracy |
| yolo26x-seg | 99.1 M | 205 MB | 395 MB | highest accuracy, very heavy |
*.pt PyTorch weights
│
│ Ultralytics model.export(format="tflite")
▼
*.onnx ◄──────────────────── ONNX intermediate (FP32)
│
├──── onnx2tf (float) ──────────────────────────────────────────┐
│ *_float32.tflite ~10 MB (full precision) │
│ *_float16.tflite ~5 MB (half precision) │
│ │
├──── onnx2tf (int8=True, calibrated on coco128) ───────────────┤
│ *_int8.tflite ~3 MB (dynamic-range) │
│ *_full_integer_quant.tflite ~3 MB (full int8) │
│ *_integer_quant.tflite ~3 MB (mixed int8 float32) │
└───────────────────────────────────────────────────────────────┘
| Technique | What it does |
|---|---|
| FP16 quantization | Halve weight precision (32→16 bits) |
| Dynamic-range INT8 | Quantize weights to int8, activations stay FP32 at runtime |
| Full-integer INT8 | Force everything to int8, no float fallback |
| Mixed quantization | Backbone INT8, detection head FP32 |
| Calibration | Representative dataset to compute activation ranges |
What we do NOT use (but could):
| Technique | Why not |
|---|---|
| QAT (Quantization-Aware Training) | Needs retraining; out of scope |
| Pruning / Clustering | More useful on larger models |
| Edge TPU / NNAPI delegates | Desktop demo, not mobile native |
| Split suppressions (WebGL) | WebGL-compatible ONNX export |
Note: Dynamic-range INT8 stores weights as int8 but dequantizes to FP32 at runtime.
Note: Full-integer INT8 forces everything to int8 : the detection head confidence scores are bad. Kept as a demo of what goes wrong when you blindly quantise without tree analysis.
Note: Full-integer INT8 is not supported for segmentation models due to Split operators in the graph (available for detect models).
┌──────────────────────────────────────────────────────────────────┐
│ BACKBONE (model.0 → model.22) │
│ Feature extraction: Conv, BatchNorm, activations │
│ Values span wide ranges → rounding to 256 int8 levels is fine. │
└──────────────────────────┬───────────────────────────────────────┘
│
│ ◄────────────── Quantizes well to INT8
▼
┌──────────────────────────────────────────────────────────────────┐
│ DETECTION HEAD (model.23) │
│ Boxes (cv2) + Classes (cv3) + NMS (TopK, Sigmoid, Gather) │
│ Confidence scores live in a narrow range: 0.30 – 0.90 │
│ │
│ INT8 = 256 levels for [0,1] → step = 0.004 │
│ conf 0.34 and conf 0.30 round to the SAME int8 value │
└──────────────────────────────────────────────────────────────────┘
Detections fall below threshold, so objects vanish.
It depends on the platform:
- Web: WebGPU (best), WebGL (fallback), WASM (universal fallback)
- Mobile: NNAPI (Android), CoreML (iOS)
- Desktop: WebGPU (best), WebGL (fallback), WASM (universal fallback) with asynchronous execution
- Edge TPU: Coral USB Accelerator
*.pt PyTorch weights
│
│ Ultralytics model.export(format="onnx")
│
├──── Yolo onnx ──────────────────────────────────────────┐
│ fp32.onnx ~10 MB │
│ opset 17, not end2end │
├──── Onnx runtime ─────────────────────────────┤
│ quant.onnx ~3 MB │
│ backbone uint8 type │
│ detection head float32 type │
└─────────────────────────────────────┘
https://tigroo.github.io/embed-ai/pwa/
Note: The end2end YOLO export includes TopK / GatherElements / Mod in the detection head ... NOT supported by WebGL :(. So we export without end2end.
Note: NMS is done in JavaScript: ~1 ms overhead.
Note: Due to Split operators in the segmentation graph, full int8 quantisation is not possible for ONNX.
Backend negotiation (fastest → safest):
- WebGPU (newest, best perf when supported)
- WebGL (mature, broad support)
- WASM (universal fallback)
Various results:
- WebGPU to manually activate on browser Firefox: about:config → dom.webgpu.enabled : True
- WebGL works → 2-5× faster than WASM on most devices
- WASM on CPU with XNNPACK
pipenv install
pipenv run python main.py| Flag | Default | Description |
|---|---|---|
--output |
output |
Directory for all generated artifacts |
--model |
yolo26n-seg |
Model name (auto-downloaded if absent) |
--summary |
summary.json |
Summary JSON filename |
--pwa-only |
False | Skip TFLite export, only ONNX for PWA |
--calibration |
None | Path to calibration YAML (optional) |
Videos are auto-discovered from resources/*.mp4.
python serve_local.py