On-device agents need 3 things: ~4GB usable RAM, prefill speed, consistent quality
helloiamleonie · x · 2026-09-25
Leonie's thread version of her AI Engineer Paris talk lays out the real constraints for on-device agent models.
Three requirements:
- Memory: an iPhone's 8GB RAM is shared with the OS and apps, leaving roughly 4GB usable — which must fit both model weights and the KV cache.
- Speed: agentic workflows involve many tool calls and reasoning steps, so speed decides whether a task takes 5 seconds or 5 minutes. Prefill matters as much as decode, since agent harnesses tend to ship huge system prompts.
- Quality: multi-step workflows need the model to be consistently good at tool calling, reasoning, and long context — not just capable on paper.
Demo: with WiFi off, she serves LFM2.5-2.6B via llama.cpp on her MacBook and drives it through the hermes agent. Calendar ops, some browser automations, and file search simply don't need frontier cloud models. Local means: data never leaves the device, local latency, no WiFi needed, zero cost — and you can still delegate to a bigger model when frontier capability is required.
More from coding & agent
- The decade of agents: simulate real customers to test your agent before launch — EXM7777 · 2026-11-11
- FluentCRM MCP Server lets you manage marketing automation from Cursor or Claude — modelcontextprotocol · 2026-11-11
- Databricks' revised Genie agent tells CEO Ali Ghodsi the unvarnished truth — TechCrunch AI · 2026-11-11
- Elvis Saravia: agents will subsume traditional software — and can even cut code bloat — omarsar0 · 2026-11-11
- Anthropic's Fable 5.2 built a professional video editor overnight, unsupervised — every · 2026-11-11
- Using a third-party inference provider with your coding harness hands it shell access to your machine — TheZachMueller · 2026-11-11