Layer-hot decoding experiment on M4 fails: negative-result spec decoding writeup

azterizm · reddit · 2026-11-17

The author tested a speculative decoding variant that rides a guessed second token through each layer's already-loaded weights on a MacBook Air M4 (MLX, Qwen3 0.6B/1.7B/4B). The ride-along works (only 2-11% overhead), but early layers guess the next token right only 1/3 of the time—almost all commas and small words—and an EAGLE-style guess head didn't help (28.9% vs 29.4%). Net: 1.2 extra tokens per pass, more than eaten by coordination overhead, running 14% slower on the 4B model. Even 60% guess accuracy would yield only 18%. Best real speedup found: llama.cpp with a DFlash2 draft model, 22% faster (31% on code). Code and results open-sourced.

Original post →

More from Infra

Infra channel →