On-device agents need 3 things: ~4GB usable RAM, prefill speed, consistent quality
helloiamleonie · x · 2026-09-25
Leonie's thread version of her AI Engineer Paris talk lays out the real constraints for on-device agent models.
Three requirements:
- Memory: an iPhone's 8GB RAM is shared with the OS and apps, leaving roughly 4GB usable — which must fit both model weights and the KV cache.
- Speed: agentic workflows involve many tool calls and reasoning steps, so speed decides whether a task takes 5 seconds or 5 minutes. Prefill matters as much as decode, since agent harnesses tend to ship huge system prompts.
- Quality: multi-step workflows need the model to be consistently good at tool calling, reasoning, and long context — not just capable on paper.
Demo: with WiFi off, she serves LFM2.5-2.6B via llama.cpp on her MacBook and drives it through the hermes agent. Calendar ops, some browser automations, and file search simply don't need frontier cloud models. Local means: data never leaves the device, local latency, no WiFi needed, zero cost — and you can still delegate to a bigger model when frontier capability is required.
Related event: Liquid AI Releases LFM2.5-2.6B and Details Its On-Device Training Recipe(10 posts)→
More from coding & agent
- GitHub ships Agentic Workflows Gallery: ready-made AI agent workflows for repo tasks — marlene_zw · 2026-09-25
- Pydantic team open-sources Monty: a Python sandbox that starts in 1ms — samuelcolvin · 2026-09-25
- Ran the Same Build on 8 Agent Platforms: Only 2 Finished, 6 Failed to Post to Slack Silently — yiddo_bhushan · 2026-09-25
- Your data stack is about to get less forgiving: agents turn stale data into wrong actions — bigdata · 2026-09-25
- Bending Spoons runs 99% of AI traffic on self-hosted open models, thanks to evals — alex_verem · 2026-09-25
- AIMessage: An Open-Source P2P Memory Mesh So AI Agents Stop Forgetting Across Machines — Accomplished-Pen-491 · 2026-09-25