mlx-vlm Multi-Turn Latency Drops to 0.25s
WesEklund · x · 2026-07-11
The author details their contribution to an open-source local AI project, reducing the multi-turn agent warm latency for mlx-vlm from roughly 72 seconds to about 0.25 seconds, marking a 290x speedup.
They explain two existing optimizations:
- Prefix caching / APC: Reuses redundant computations in long contexts.
- KV cache quantization: Lowers VRAM usage, enabling larger models to run on Mac.
Previously, these two conflicted—enabling KV quantization essentially broke prefix caching. This limitation has now been resolved, allowing developers to leverage both multi-turn speed and memory efficiency simultaneously. The author mentions the new version will be released "very soon."
Related event: Cache Fix Reduces mlx-vlm Agent Latency by 290x(4 posts)→
More from coding & agent
- AI agents are starting to strain code hosting platforms — craigsdennis · 2026-07-21
- Async OPD distillation doubles throughput while matching synchronous math accuracy — _lewtun · 2026-07-21
- Omnigent 0.6.0 adds Claude Code imports, Slack approvals and desktop apps — matei_zaharia · 2026-07-21
- Google appears to have quietly shipped Gemini 3.6 Flash, with lower pricing and better agentic scores — xiaohu · 2026-07-21
- Open-source CLI audits AI tools, MCP configs, and agent skills on local machines — Initial-Copy332 · 2026-07-21
- Coding agents feel less stressful when the 5-hour limits are temporarily removed — iamrobotbear · 2026-07-21