Qwen3.8-27B in native 8-bit hits 37-55 tok/s on Apple Silicon, avoiding the 4-bit reasoning cliff

SnooPredictions515 · reddit · 2026-09-21

The author extended the Splash engine (a compiled C++/Metal speculative decoding engine by Incoai) to run Qwen3.8-27B in native 8-bit on Apple Silicon (M5 Pro, 64GB), sustaining 37-55 tok/s with zero quantization degradation.

Key findings:

Reasoning cliff: Upstream Splash's 4-bit models run fast (60 tok/s) but aggressive quantization degrades competition-grade math and multi-step derivations.
Benchmarks: Native 8-bit (Splash-HQ, 27GB) averages 36.9 tok/s across five task domains—3.73x the 9.9 tok/s MLX/llama.cpp AR baseline—and hits 54.8 tok/s on math. It beats MTPLX-Q8 on identical weights by +39% overall (+92% math) due to the compiled Metal backend's lower dispatch overhead.
Precision-speed paradox: Uncompressed 8-bit (36.9 tok/s) slightly beat compressed 8-bit (36.5 tok/s)—compression flattens logits and lowers draft acceptance; sharper native-8-bit logits mean fewer verification rollbacks and higher net throughput.
Context scaling tested up to 256k, with live telemetry to 190k.

Weights, the Q8 Metal runtime fork, and full benchmark JSON logs are open-sourced (Hugging Face / GitHub). The fork adds schema 5 loading and compiled Metal Q8 tiled decode kernels while staying 100% backward compatible with upstream Q4 models.

Original post →

Qwen3.8-27B in native 8-bit hits 37-55 tok/s on Apple Silicon, avoiding the 4-bit reasoning cliff

More from Infra