Hand-rolled C+Metal runtime pushes Qwen3.8-27B to 18 tok/s on a 36GB M3 Pro Mac
buryhuang · reddit · 2026-08-19
The author runs Qwen3.8-27B locally on a 36GB M3 Pro at 18 tok/s — daily-driver territory. After testing llama.cpp, mlx-lm, and oMLX, he went back to his own C + Metal runtime that compiles the model into a machine-specific image: Q4 weights (15GB) mmap'd into unified memory, fronted by a thin OpenAI-compatible API.
The weekend's focus was speculative decoding: replay, verification kernels, and draft vocabulary. Results:
- Full LRUCache implementation (1,155 output tokens) averaged 17.7 tok/s end-to-end; six runs produced identical output at 55ms inter-token latency;
- Prose is slower (lower draft acceptance), 10-11 tok/s; five tasks averaged 11.95 tok/s vs 7.96 without speculative decoding — roughly 1.5x speedup;
- TTFT: 1.4s short prompt, 2.7s for 128 tokens.
He notes the M3 Pro's 120GB/s bandwidth means 18 tok/s at 27B Q4 is close to the bandwidth wall, and his runtime is currently clearly faster than llama.cpp on the same machine. One layer-fusion kernel remains. The project is open source (MIT).
More from coding & agent
- Dev of Private Software Like Cooking Art; Agent Trend to Redefine Perception — pixlpa · 2026-08-24
- Agentic coding accessibility will reshape understanding of software complexity — pixlpa · 2026-08-24
- Devin Agent bypasses Slack block by finding emails in git logs — sandylikesfrogs · 2026-08-24
- Developer habits shift: Agents become collaborators from simple tools — latticecut · 2026-08-24
- Dev bottleneck shifts from writing to reading code: exe.dev co-founder — thursdai_pod · 2026-08-24
- The biggest AI mistake: trying to reinvent the wheel instead of using tools — Tired40s · 2026-08-24