InferNEON runs MoE LLMs on dirt-cheap boards with a bare-metal Rust kernel and ternary experts
paraschopra · x · 2026-10-27
Resident Karthik's InferNEON pushes LLMs onto ultra-cheap embedded hardware. Three angles: (1) a bare-metal Rust inference kernel (no OS, no libc, hand-written ARM NEON, nostd) already running OLMoE-1B-7B in QEMU, matching Hugging Face token-for-token; (2) replay simulations with real router traces showing raw capacity beats clever expert caching on slow storage; (3) an 'eNPU' — a cheap Ethernet-attached board holding ternary experts resident with only activations crossing the wire, turning experts from memory-bound to compute-bound on a Pi CM4 (4.5x faster than bf16, 0.25ms per layer round trip). Goal: serious local AI on hardware cheaper than a dinner out.
More from coding & agent
- Compound Engineering 3.33.0 ships PR workflow upgrades to curb scope-creeping agents — kieranklaassen · 2026-10-27
- Lovable invokes 753.6B Cloudflare Workers monthly, migrated from Next.js to TanStack Start — ritakozlov · 2026-10-27
- Harness Lens now indexes 30 open source coding harnesses and accepts new submissions — yb2698 · 2026-10-27
- Prompting agents at the grammar level: pattern language as interface design — DanGrover · 2026-10-27
- Microsoft Foundry hosted agents get mailboxes, identity and org-chart presence — marlene_zw · 2026-10-27
- 36 blind AI judges pick a solo-session agent setup over Superpowers 36-0 — Kai_ThoughtArchitect · 2026-10-27