Running a 28.9M Parameter LLM on an $8 Microcontroller at 9 Tokens/s
Saboo_Shubham_ · x · 2026-07-30
A developer released esp32-ai, an open-source project that successfully runs a 28.9M parameter LLM locally on an ESP32-S3 microcontroller costing about $8.
- Performance: Runs entirely offline, generating text at roughly 9 tokens/s, and outputs directly to a wired tiny screen.
- Technical Breakthrough: Compared to previous 260K parameter models on similar chips, this is a hundredfold leap. The core optimization leverages Per-Layer Embeddings from Google Gemma, storing the vast majority of parameters (25M) in flash memory rather than RAM.
Related event: 8-Dollar ESP32-S3 Microcontroller Runs 28.9M-Parameter LLM(3 posts)→
More from Infra
- Mac Inference Speed Surges 80% as Open Source Community Breaks Performance Limits — gajesh · 2026-07-30
- Running Text Embedding Models on Jetson Nano for RAG — S_Anv · 2026-07-30
- Sentinel Framework Repurposes Old Android Phones into LAN-based AI Vision Nodes — tom_doerr · 2026-07-30
- Agents Reshape LLM Workloads: Input-Output Token Ratio Hits 300:1 — appenz · 2026-07-30
- Qualcomm Whitepaper: AI Device Penetration Exceeds 66%, Moving Towards Distributed Personal AI — 量子位 · 2026-07-30
- Community Optimizes Kimi Inference on AMD MI355X to Beat NVIDIA B200 — a1zhang · 2026-07-30