270M Small Model Runs Agents on Mobile
ivan_digital · reddit · 2026-07-19
The author maintains the open-source project speech-android, implementing a voice agent running entirely on an Android device: VAD → streaming STT → tool-calling LLM → device actions → TTS response. The entire pipeline uses only 1.2 GB RAM and can start responding about 0.9 seconds after speech ends on a Galaxy S23 Ultra.
They believe the keys to making small models work reliably as agents are twofold:
- Fine-tuning using a 9.5 MB LoRA adapter based on exact tool schemas and device state sequences, rather than generalized function-calling training;
- Only exposing legal tools for the current state at runtime, as narrowing the action space is more effective than prompt engineering.
They also emphasized that the routing has no keyword/regex fallbacks; tool selection is entirely decided by the model. A single tool call takes an average of 294 ms.
More from coding & agent
- Soft Clamp cuts tool-call overuse in multi-teacher distillation, from 13.7% to 9.0% — antgroup · 2026-07-21
- Agent harness memory loss and compaction are still a major usability problem — adityaag · 2026-07-21
- SpecJudge runs locally on Ollama to pick the right-sized AI model for your project — jokiruiz · 2026-07-21
- A developer maps out six design rules for CLIs that humans and AI agents can both use — yujiezha · 2026-07-21
- A coding-agent skill that forces ADHD-friendly, answer-first output — ayghri · 2026-07-21
- A set of agent skills for CAD, robotics, and hardware design — earthtojake · 2026-07-21