Hand-rolled C+Metal runtime pushes Qwen3.8-27B to 18 tok/s on a 36GB M3 Pro Mac
buryhuang · reddit · 2026-08-19
The author runs Qwen3.8-27B locally on a 36GB M3 Pro at 18 tok/s — daily-driver territory. After testing llama.cpp, mlx-lm, and oMLX, he went back to his own C + Metal runtime that compiles the model into a machine-specific image: Q4 weights (15GB) mmap'd into unified memory, fronted by a thin OpenAI-compatible API.
The weekend's focus was speculative decoding: replay, verification kernels, and draft vocabulary. Results:
- Full LRUCache implementation (1,155 output tokens) averaged 17.7 tok/s end-to-end; six runs produced identical output at 55ms inter-token latency;
- Prose is slower (lower draft acceptance), 10-11 tok/s; five tasks averaged 11.95 tok/s vs 7.96 without speculative decoding — roughly 1.5x speedup;
- TTFT: 1.4s short prompt, 2.7s for 128 tokens.
He notes the M3 Pro's 120GB/s bandwidth means 18 tok/s at 27B Q4 is close to the bandwidth wall, and his runtime is currently clearly faster than llama.cpp on the same machine. One layer-fusion kernel remains. The project is open source (MIT).
More from coding & agent
- Gemini 4 Argon goes live in Antigravity with its own weekly quota for Pro subscribers — rseroter · 2026-10-16
- LangChain revamps human-in-the-loop middleware with typed schemas for tool interrupts — Hacubu · 2026-10-16
- Robot fully controlled by frontier LLMs assembles gadgets autonomously for 1.5 hours error-free — ihorbeaver · 2026-10-16
- Why the future of agents is basically Node and JavaScript, in one argument — verrsane · 2026-10-16
- Vibe Coding a Data Pipeline Is Easy; Running It at Billions of Rows Is Not — Zachly · 2026-10-16
- How Claude Code Review works at GA: admin setup, auto reviews on every new PR — ClaudeDevs · 2026-10-16