Qwen3.8-27B in native 8-bit hits 37-55 tok/s on Apple Silicon, avoiding the 4-bit reasoning cliff
SnooPredictions515 · reddit · 2026-09-21
The author extended the Splash engine (a compiled C++/Metal speculative decoding engine by Incoai) to run Qwen3.8-27B in native 8-bit on Apple Silicon (M5 Pro, 64GB), sustaining 37-55 tok/s with zero quantization degradation.
Key findings:
- Reasoning cliff: Upstream Splash's 4-bit models run fast (60 tok/s) but aggressive quantization degrades competition-grade math and multi-step derivations.
- Benchmarks: Native 8-bit (Splash-HQ, 27GB) averages 36.9 tok/s across five task domains—3.73x the 9.9 tok/s MLX/llama.cpp AR baseline—and hits 54.8 tok/s on math. It beats MTPLX-Q8 on identical weights by +39% overall (+92% math) due to the compiled Metal backend's lower dispatch overhead.
- Precision-speed paradox: Uncompressed 8-bit (36.9 tok/s) slightly beat compressed 8-bit (36.5 tok/s)—compression flattens logits and lowers draft acceptance; sharper native-8-bit logits mean fewer verification rollbacks and higher net throughput.
- Context scaling tested up to 256k, with live telemetry to 190k.
Weights, the Q8 Metal runtime fork, and full benchmark JSON logs are open-sourced (Hugging Face / GitHub). The fork adds schema 5 loading and compiled Metal Q8 tiled decode kernels while staying 100% backward compatible with upstream Q4 models.
More from Infra
- Qdrant experiments: 10→500 candidate depth lifts best-possible nDCG by 0.28 but real score by ≤0.01 — qdrant_engine · 2026-09-21
- gemini-cli PR Fixes Process Hang on Exit via stdin and MCP Child Process Cleanup — Pcmhacker-piro · 2026-09-21
- Daniel Lemire Tests Whether CPUs Can Take More Than One Branch Per Cycle — lemire · 2026-09-21
- Google's Orphaned VMs Patches Keep VMs Running While Host Kernel Goes Offline — jedisct1 · 2026-09-21
- djev-run Deploys DiffusionGemma on Cloud Run's RTX PRO 6000 Blackwell for ~$3/hr — bodonoghue85 · 2026-09-21
- 16GB (often 12GB) is the realistic VRAM ceiling for most people running local AI — ECrispy · 2026-09-21