Running a 2M-param embedding model on an ESP32 with no_std Rust at 830 embeddings/s
qdrant_engine · x · 2026-11-17
Qdrant engineer Clelia ported Minish Lab's potion-base-2M embedding model to an ESP32 microcontroller with only 16MB of memory.
- Written in nostd Rust: a build script compiles model weights and vocabulary into constants and a blob baked into the firmware, so the chip only reads memory at runtime
- Throughput: 1.2ms to embed a dozen tokens, roughly 830 embeddings/s on short texts — far below the 11-15k embeddings/s static models reach on a 48GB Mac, but using 3000x less memory
- Takeaway: search can genuinely run on the edge
More from Infra
- Engineer runs Gemma 3 270M entirely inside an office laser printer at 60s per token — osanseviero · 2026-11-17
- Older GPUs abandoned by new inference engines: are they still worth keeping? — pipecanon · 2026-11-17
- Blogger argues total compute controls create blind spots and accelerate adversaries — TinfoilTricorn · 2026-11-17
- How rollout staleness hurts on-policy distillation, and how to build a faster pipeline — burkov · 2026-11-17
- SK hynix's Nvidia revenue doubled to $24B, but US customer concentration is the real risk — tengyanAI · 2026-11-17
- macOS TCP Stack Breaks After Exactly 49.7 Days of Uptime Due to 32-bit Timestamp Overflow — DanielLockyer · 2026-11-17