mlx-vlm Multi-Turn Latency Drops to 0.25s
WesEklund · x · 2026-07-12
The author notes their first major contribution to open-source local AI: reducing the multi-turn agent warm latency of mlx-vlm from roughly 72 seconds down to 0.25 seconds, an improvement of about 290x.
Background
- Previously, prefix caching and KV cache quantization could not be enabled simultaneously.
- This meant users had to choose between "faster multi-turn dialogue" and "lower memory usage to fit the model on a Mac."
The Improvement
- By making the two compatible, mlx-vlm can simultaneously retain long-context reuse capabilities and a smaller memory footprint.
- The author emphasizes that this is a massive unlock for agentic workflows.
Conclusion
- This optimization is geared toward local AI and multi-turn reasoning workflows.
- The author also mentioned it will be included in an upcoming release very soon.
Related event: Cache Fix Reduces mlx-vlm Agent Latency by 290x(4 posts)→
More from coding & agent
- HeyGen adds a media-sourcing skill for coding agents with 75k images and 10k tracks — HeyGen · 2026-07-22
- Agent search bottlenecks are now about variance, not raw latency — rohanpaul_ai · 2026-07-22
- LangSmith adds tracing for Pipecat, LiveKit, OpenAI Realtime, and Gemini Live — LangChain · 2026-07-22
- An MCP server signs every AI agent tool call into a verifiable Merkle chain — Funky_Chicken_22 · 2026-07-22
- Annotated transcript of a Claude Code team interview is now available — trq212 · 2026-07-22
- Claude Code skill uses 10 Markdown rules to make outputs ADHD-friendly — alex_verem · 2026-07-22