M5 Ultra 96GB local inference: 3,200 tok/s aggregate prefill over 112M tokens with custom MLX server
Every-Fortune-3151 · reddit · 2026-09-24
A Redditor ran Qwen 3.8 FN (4/8-bit mixed quant) locally on a base M5 Ultra 96GB with a custom MLX server, processing 112M tokens of agentic coding work: 3,200 tok/s aggregate prefill and 170 tok/s decode at 4-way concurrency, single-stream median prefill 828 tok/s. With a Qwen 27b orchestrator on a PC and 512k total context (8-bit KV, 90GB RAM), prompt caching hit 89% across 1,700 sub-agent calls. The AI-built server adds GDN state pinned in GPU registers, fused HyperConnection+RMSNorm, MoE tile compaction discarding 98.3% zero-padding, and an FP16 cache of top-64 hot experts (960MB).
More from coding & agent
- Two Codex agents coded, tested and submitted iOS & Android apps to stores in parallel — burkov · 2026-09-24
- You misunderstand AGENTS.md: how coding tools read project instructions — dotey · 2026-09-24
- Forced fresh-worker swap at 60k tokens: 10/12 Terminal-Bench tasks still pass — key_of_door · 2026-09-24
- Solo Dev Open-Sources Tapioca, a Go Terminal Coding Agent That Runs Fully Local — Practical_Witness_95 · 2026-09-24
- Anthropic launches Claude Marketplace with 2,000+ connectors and third-party agents — claudeai · 2026-09-24
- Nyx Terminal launches for Mac: a terminal built for running many coding agents at once — henrymodis · 2026-09-24