M5 Ultra 96GB local inference: 3,200 tok/s aggregate prefill over 112M tokens with custom MLX server
Every-Fortune-3151 · reddit · 2026-09-24
A Redditor ran Qwen 3.8 FN (4/8-bit mixed quant) locally on a base M5 Ultra 96GB with a custom MLX server, processing 112M tokens of agentic coding work: 3,200 tok/s aggregate prefill and 170 tok/s decode at 4-way concurrency, single-stream median prefill 828 tok/s. With a Qwen 27b orchestrator on a PC and 512k total context (8-bit KV, 90GB RAM), prompt caching hit 89% across 1,700 sub-agent calls. The AI-built server adds GDN state pinned in GPU registers, fused HyperConnection+RMSNorm, MoE tile compaction discarding 98.3% zero-padding, and an FP16 cache of top-64 hot experts (960MB).
More from coding & agent
- Rivet launches SQLite Forking: 20ms agent-thread forks with zero data copying — samgoodwin89 · 2026-11-11
- YouArt Desktop launches: AI agent edits video in CapCut, DaVinci and Premiere from client feedback — mhdfaran · 2026-11-11
- Boubou: one-command local voice AI that serves your phone and robots over Wi-Fi — LuozhuZhang · 2026-11-11
- Python 3.15 makes AttributeError actionable: suggests .append for .push mix-ups — KhuyenTran16 · 2026-11-11
- sentence-transformers maintainer releases new embedding pooling training method, code & checkpoints — tomaarsen · 2026-11-11
- droid-mcp turns your Android phone into an MCP server with 160+ tools for Claude Code — zvone1122 · 2026-11-11