Running 28.9M LLM on $8 ESP32: 10 tok/s Fully On-Device
FinanceYF5 · x · 2026-08-06
A developer has successfully run a Large Language Model (LLM) completely locally on an $8 ESP32-S3 microcontroller to create a barista AI.
Core Technical Specs:
- Parameters: 28.9M (25M stored in a flash lookup table)
- Hardware: ESP32-S3 (512KB SRAM, 8MB PSRAM, 16MB flash)
- Speed: 9.88 tokens/s end-to-end, 94.9 ms/token compute
- Model Size: 14.9MB at 4-bit quantization
Implementation:
Given the limited fast memory of microcontrollers, the developer utilized Per-Layer Embeddings, an idea from Google's Gemma 3n. By offloading the majority of model weights to flash memory and keeping only frequently touched activations and norm weights in SRAM, the project bypasses memory bottlenecks to achieve fully offline, GPU-free edge AI inference.
More from Infra
- Nebius Inference Platform Hits Milestone in Artificial Analysis Accuracy Index — demian_ai · 2026-08-06
- Cloudflare Emerging as the Agent Cloud via Rapid Iteration and Weird Bets — threepointone · 2026-08-06
- How Open-Source AI Inference Became Critical Infrastructure: a16z Podcast — a16z Podcast · 2026-08-06
- PlanetScale Details Massively Parallel Backups: Restoring PB-Scale Databases at 50GB/s — bibryam · 2026-08-06
- Calling for Decentralized AI: From the Cloud Back to a $5K Local Personalized AGI — Dan_Jeffries1 · 2026-08-06
- Maple 20B Hits 9,885 tokens/s on a Single NVIDIA GH200 in Concurrency Test — MaziyarPanahi · 2026-08-06