Layer-hot decoding experiment on M4 fails: negative-result spec decoding writeup
azterizm · reddit · 2026-11-17
The author tested a speculative decoding variant that rides a guessed second token through each layer's already-loaded weights on a MacBook Air M4 (MLX, Qwen3 0.6B/1.7B/4B). The ride-along works (only 2-11% overhead), but early layers guess the next token right only 1/3 of the time—almost all commas and small words—and an EAGLE-style guess head didn't help (28.9% vs 29.4%). Net: 1.2 extra tokens per pass, more than eaten by coordination overhead, running 14% slower on the 4B model. Even 60% guess accuracy would yield only 18%. Best real speedup found: llama.cpp with a DFlash2 draft model, 22% faster (31% on code). Code and results open-sourced.
More from Infra
- TensorFold Runs Local LLMs at 874 tok/s With Native Metal/CUDA Inference — AIFlow_ML · 2026-11-17
- AMD's Dan McNamara: billions of AI agents will reshape CPU demand, Venice chips now shipping — firstadopter · 2026-11-17
- User Slams Nvidia for Capping VRAM at 128GB While Claiming to Push Local AI — AIFlow_ML · 2026-11-17
- Commerce Dept and DOJ issue joint statement on Megaspeed investigation — ShakeelHashim · 2026-11-17
- Cloudflare open-sources Kitesurf, a Chromium-free browser engine for AI agents — shoumikgoswami · 2026-11-17
- Biggest U.S. data centers emit 25.7M tons of CO2 a year, rivaling entire nations — asusarla · 2026-11-17