mlx-vlm Multi-Turn Latency Drops to 0.25s
WesEklund · x · 2026-07-11
The author details their contribution to an open-source local AI project, reducing the multi-turn agent warm latency for mlx-vlm from roughly 72 seconds to about 0.25 seconds, marking a 290x speedup.
They explain two existing optimizations:
- Prefix caching / APC: Reuses redundant computations in long contexts.
- KV cache quantization: Lowers VRAM usage, enabling larger models to run on Mac.
Previously, these two conflicted—enabling KV quantization essentially broke prefix caching. This limitation has now been resolved, allowing developers to leverage both multi-turn speed and memory efficiency simultaneously. The author mentions the new version will be released "very soon."
Related event: Cache Fix Reduces mlx-vlm Agent Latency by 290x(4 posts)→
More from coding & agent
- GPT-6 Astra beats Factorio with enemies in 44 in-game hours at ~$4,500 API cost — liminal_bardo · 2026-09-11
- Treating agents like 50 First Dates: a 3-layer context system so every conversation doesn't start from zero — evielync · 2026-09-11
- SmolVM open-sources persistent computer infrastructure for agents that outlive chat sessions — aniketmaurya · 2026-09-11
- ARRM targets silent economic regressions in AI agents that functional tests miss — Beautiful_Belt_601 · 2026-09-11
- Dev builds browser 3D pizza delivery game with Claude: physics, GPS pathfinding, traffic AI — vinishkapoor · 2026-09-11
- Build X Carousel Posts from One Wide Image: A Splitter Tool Plus YouMind Skill Workflow — sujingshen · 2026-09-11