Running a 28.9M Parameter LLM on an $8 Microcontroller at 9 Tokens/s
Saboo_Shubham_ · x · 2026-07-30
A developer released esp32-ai, an open-source project that successfully runs a 28.9M parameter LLM locally on an ESP32-S3 microcontroller costing about $8.
- Performance: Runs entirely offline, generating text at roughly 9 tokens/s, and outputs directly to a wired tiny screen.
- Technical Breakthrough: Compared to previous 260K parameter models on similar chips, this is a hundredfold leap. The core optimization leverages Per-Layer Embeddings from Google Gemma, storing the vast majority of parameters (25M) in flash memory rather than RAM.
Related event: 8-Dollar ESP32-S3 Microcontroller Runs 28.9M-Parameter LLM(3 posts)→
More from Infra
- Neocloud Stocks Slashed Again: CRWV -10%, NBIS -9%, IREN -8% in One Day — JOBhakdi · 2026-10-16
- Neocloud stocks sink on Gartner's "80% will fail" report; analyst calls it garbage — McDonaghMatthew · 2026-10-16
- JapanFold Claims One Tenstorrent Server Beats an Nvidia DGX B200 on Bio Models at a Third of the Price — DavidBennett__ · 2026-10-16
- vLLM ships Day-0 RL support for Beam in SkyRL, +62% LoRA decode throughput — vllm_project · 2026-10-16
- DFlash2 speeds Beam decode up to 3.03× across benchmarks, vLLM reports — vllm_project · 2026-10-16
- Inside Beam: 501B params, only 23B active per token via 8-of-256 expert routing — vllm_project · 2026-10-16