mlx-vlm Warm Start Latency Slashed by 290x
WesEklund · x · 2026-07-11
The author shares their first major contribution to an open-source local AI project: reducing the multi-turn agent warm latency of mlx-vlm from roughly 72 seconds down to 0.25 seconds, achieving approximately a 290x improvement.
They note the project already featured two key optimization techniques:
- Prefix caching / APC: Reuses redundant computations across long contexts.
- KV cache quantization: Compresses runtime cache, allowing larger models to run on Mac.
The core of this optimization was better integrating these mechanisms to drastically improve the first-turn/warm-start experience during multi-turn interactions.
Related event: Cache Fix Reduces mlx-vlm Agent Latency by 290x(4 posts)→
More from coding & agent
- alphaXiv open-sources OpenResearch to run parallel research agents with any model — alphaXiv · 2026-09-11
- MathModelAgent gains traction: auto-solves math modeling and writes a submission-ready paper — jihe520 · 2026-09-11
- DeskcommCRM: open-source AI sales CRM with native agents and WhatsApp hits 1k stars — melgarafael · 2026-09-11
- hyperresearch: agent-driven knowledge base that turns web research into a searchable wiki — jordan-gibbs · 2026-09-11
- Forter's 13 lessons from its agent sprint: skip custom RAG, lean on mature enterprise search — bibryam · 2026-09-11
- Two real 'company brains' opened up live: Gorgias' in-house Cortex vs Slite — femke_plantinga · 2026-09-11