Running an LLM on an ESP32 with Only 81KB of Memory
Similar_Wealth_1850 · reddit · 2026-08-09
A developer successfully ran a micro-LLM on an ESP32 dev board with only 512KB of SRAM, drawing significant attention from the community.
Technical Details:
- Architecture: A 5.2M parameter Mixture-of-Experts (MoE) model with 16 experts, quantized to INT4.
- Memory Optimization: The engine streams experts from Flash to SRAM, running only one expert per token. This uses just 81KB of memory, leaving 215KB for the KV Cache.
- Model Size: Features 6 layers, 4 heads, and 128 embedding dimensions. Quantized weights are only 3.1MB.
- Performance: Achieves around 5 Tokens/s on an ESP32 Dev Kit V1.
The developer also added a Math Harness to help solve simple equations and implemented Attention Sink to improve the context window experience. The project is open-source on GitHub.
More from Infra
- MLCommons launches MLPerf Endpoints, a benchmark that measures inference endpoints the way buyers actually purchase them — TheKanter · 2026-10-16
- Neocloud M&A decoded: CoreWeave, Nebius, Crusoe, Nscale all bet margins vanish beyond GPU scarcity — AccBalanced · 2026-10-16
- AI-Driven Timing-Aware Power Optimization Uses Learned Ranking With Baseline-Preserving Repair — Amit Sharma · 2026-10-16
- Nscale AI datacenter plagued by coolant leaks, transformer fire, 10 safety violations — saranormous · 2026-10-16
- WSL 3.0 ships with native containers and Linux 6.18 kernel, syscall overhead down ~16% — solyarisoftware · 2026-10-16
- Are 22 GB modded RTX 2080 Ti cards a viable budget local LLM option? — Dave86ch · 2026-10-16