glance-vlm speedlab goes open source: MLX 8-bit cuts local camera VLM latency 27.6%
natesiggard · x · 2026-09-24
Yohei Nakajima open-sourced glance-vlm speedlab, a measurement-first latency study that turns any webcam into multiple locally running live AI detectors (emotion, count, object).
Key results (Apple M5, 32GB)
- An accepted 8-bit MLX direct scorer cut fresh-frame p50 from 358.5ms to 259.6ms (1.381×, 27.6% lower latency) while matching 84/84 fixed-suite decisions with max probability drift of 0.039
- Native multi-question batching gave 2.405× speedup; a smaller 2B model was 2.994× faster but failed the quality guardrail
- 21 preregistered experiments with reproducible benchmarks, a paper, and documented failures; live single-question loop runs 210ms (PyTorch/MPS FP16) or 160ms (MLX 8-bit)
Methodological takeaway: optimize realized model work, not configuration labels or proxy token counts—several attractive changes produced no safe end-to-end win.
More from Infra
- Wasmer's Swift SDK brings sandboxed Python, Node.js and FFmpeg running fully on-device on iOS — jedisct1 · 2026-09-24
- Read-only MCP servers for Proxmox and pfSense: agents hit the API, not screenshots — Mustela__ · 2026-09-24
- From one GPU to millions of users: lessons in LLM inference system design — metalvendetta · 2026-09-24
- Modal Labs in Talks to Raise at $15 Billion Valuation as Inference Rush Heats Up — nmasc_ · 2026-09-24
- Google paper: 55-70% of quantized LLM cold-start latency is just model loading — rohanpaul_ai · 2026-09-24
- Optical computing for AI debated as veteran cites lack of European interest — IgorCarron · 2026-09-24