mlx-vlm Warm Start Latency Slashed by 290x
WesEklund · x · 2026-07-11
The author shares their first major contribution to an open-source local AI project: reducing the multi-turn agent warm latency of mlx-vlm from roughly 72 seconds down to 0.25 seconds, achieving approximately a 290x improvement.
They note the project already featured two key optimization techniques:
- Prefix caching / APC: Reuses redundant computations across long contexts.
- KV cache quantization: Compresses runtime cache, allowing larger models to run on Mac.
The core of this optimization was better integrating these mechanisms to drastically improve the first-turn/warm-start experience during multi-turn interactions.
Related event: Cache Fix Reduces mlx-vlm Agent Latency by 290x(4 posts)→
More from coding & agent
- DeepMind paper: LLMs can't sample randomly on their own — pass random numbers from code instead — rohanpaul_ai · 2026-10-27
- BELL: Open-Source Stack Combines Browserbase, Eve, iMessage and Stripe Link for Personal Agents — jeff_weinstein · 2026-10-27
- Legora runs 2,000 parallel agent evals on Modal Sandboxes for legal AI — blelbach · 2026-10-27
- Two engineers rebuilt a FedEx supplier's billing core in 3.5 months vs an estimated 7-8 — alex_verem · 2026-10-27
- GPT-6.1 Terra scores 48 on AA Coding Agent Index at one-third the cost of GPT-5.6 Terra — ArtificialAnlys · 2026-10-27
- Shift-1 router cuts coding agent spend up to 55% while keeping SWE-bench accuracy — ypatil125 · 2026-10-27