Fully Offline: Running a Custom Q/A Model on ESP32S3 Microcontroller
slvDev_ · reddit · 2026-08-03
A developer successfully ran a small Q/A model (barista v0.1) for espresso troubleshooting fully offline on an ESP32S3 N16R8 microcontroller. Users can type a question via USB, and the device streams the answer to an OLED display or terminal.
Core Technical Implementations:
- Memory Mapping: Large PLE and token-embedding tables stay memory-mapped in flash, as the model only needs one row from each table at a given position.
- Asymmetric Vocabulary: The model reads an 8,000+ input-token vocabulary but writes only 854 output classes. This reduces the output head from about 1M parameters to 109K, which is sufficient for narrow-domain answers.
The author notes that because the model physically lacks digit characters, it cannot emit numbers, and unrelated queries might yield bad advice rather than a refusal. The current focus is on gathering more high-quality Q/A data rather than just adding layers.
More from Infra
- Minimax H3 Tested: Runs Locally on 8GB VRAM — inuptia · 2026-08-04
- Compute Scarcity vs. Creativity: Debating the Future of Neo AI Labs — reneeshah123 · 2026-08-04
- Self-Improving Agents Optimize vLLM, Boosting DeepSeek Throughput by 16% — yisongyue · 2026-08-04
- NVIDIA and KAIST Launch Joint AI Lab to Advance Agentic AI in Korea — hyunw_kim · 2026-08-04
- Running Frontier Models on 24GB VRAM: Local Deployment Challenges Cloud — mintybadgerme · 2026-08-04
- Self-Hosting AI Dev Environments: Sandboxing and Multi-Model Orchestration — Illhoon · 2026-08-04