Optimizing Local Concurrent Inference on Mac Studio
marzukia · reddit · 2026-07-18
The author continues to optimize a forked local inference engine, qMLX, aiming to replace parallelism with concurrency on a 96GB Mac Studio.
The core result: in a 20-minute test with 3 concurrent sessions, the system received 789,351 prompt tokens but only recomputed 48,996 on the GPU. About 93.8% of the tokens were retrieved directly from the on-disk KV cache, which the author says effectively slashes the prefill compute by roughly 1/16.
Two key changes enabled this:
- Implemented partial persistent storage for the Gated DeltaNet KV cache
- Developed a more rational eviction strategy
Having mostly removed the hot cache path, qMLX has evolved into an inference engine that "recovers state from SSD and processes requests serially," and the author has started using it as a local worker for a frontier orchestrator. The author believes this "concurrency without parallelism" approach finally allows him to run sub-agents on local models.
More from coding & agent
- GPT-6 Astra beats Factorio with enemies in 44 in-game hours at ~$4,500 API cost — liminal_bardo · 2026-09-11
- 105 hidden bugs, 2 repos: DeepSeek V4.1 Flash fixes 24 at $1.80 vs Opus 5's 27 at $51.33 — ChartsJournalX · 2026-09-11
- Investment Analyst Asks How to Build a Claude-Based Diligence Agent Stack — Careless_Tie2286 · 2026-09-11
- Treating agents like 50 First Dates: a 3-layer context system so every conversation doesn't start from zero — evielync · 2026-09-11
- Running the Firefox MCP on Android via Termux, ngrok, and mcp-proxy — Nervous-Strain7544 · 2026-09-11
- SmolVM open-sources persistent computer infrastructure for agents that outlive chat sessions — aniketmaurya · 2026-09-11