Running an LLM on an ESP32 with Only 81KB of Memory
Similar_Wealth_1850 · reddit · 2026-08-09
A developer successfully ran a micro-LLM on an ESP32 dev board with only 512KB of SRAM, drawing significant attention from the community.
Technical Details:
- Architecture: A 5.2M parameter Mixture-of-Experts (MoE) model with 16 experts, quantized to INT4.
- Memory Optimization: The engine streams experts from Flash to SRAM, running only one expert per token. This uses just 81KB of memory, leaving 215KB for the KV Cache.
- Model Size: Features 6 layers, 4 heads, and 128 embedding dimensions. Quantized weights are only 3.1MB.
- Performance: Achieves around 5 Tokens/s on an ESP32 Dev Kit V1.
The developer also added a Math Harness to help solve simple equations and implemented Attention Sink to improve the context window experience. The project is open-source on GitHub.
More from Infra
- Nvidia's Rubin Ultra Shifts from HBM to Optical Interconnects, Altering Market Dynamics — zephyr_z9 · 2026-08-09
- Running SD Natively on Android: SDXL Takes 20 Minutes on a Phone — Silent-Paramedic4063 · 2026-08-09
- LFM 2.6B Hits 260 Tokens/s on RTX 3090: A Dev's Hands-On Review — Borkato · 2026-08-09
- Report: Nvidia to Invest Up to $3B in AI Data Center Power Developer Lancium — rohanpaul_ai · 2026-08-09
- Intel Xeon LLM Bandwidth Halved? Local MoE Inference Tuning Log — GetOutOfMyFeedNow · 2026-08-09
- Ryan Dahl Launches celld: Self-Hosted Distributed Durable Objects Using SQLite — steren · 2026-08-09