mlx-vlm Multi-Turn Latency Drops to 0.25s
WesEklund · x · 2026-07-11
The author details their contribution to an open-source local AI project, reducing the multi-turn agent warm latency for mlx-vlm from roughly 72 seconds to about 0.25 seconds, marking a 290x speedup.
They explain two existing optimizations:
- Prefix caching / APC: Reuses redundant computations in long contexts.
- KV cache quantization: Lowers VRAM usage, enabling larger models to run on Mac.
Previously, these two conflicted—enabling KV quantization essentially broke prefix caching. This limitation has now been resolved, allowing developers to leverage both multi-turn speed and memory efficiency simultaneously. The author mentions the new version will be released "very soon."
Related event: Cache Fix Reduces mlx-vlm Agent Latency by 290x(4 posts)→
More from coding & agent
- Sol61 flunks instruction-following: skips mandated research even when explicitly told — lucasmeijer · 2026-10-26
- Ben Bajarin: Frontier AI is becoming the enterprise cyber-defense stack — BenBajarin · 2026-10-26
- Building AI bots on raw APIs? Devs say models are 'dumb without the harness' — madeofAjala · 2026-10-26
- HarnessMM cuts agent tokens per task by up to 88% while raising success rates — omarsar0 · 2026-10-26
- Microsoft demos intent-classifying agents built with Copilot App and Foundry, deployable across Teams and Outlook — marlene_zw · 2026-10-26
- Looking for a wearable mic with best battery life that feeds recordings to personal agents via MCP — altryne · 2026-10-26