Systolic arrays make speculative decoding latency a staircase synced to array width
teortaxesTex · x · 2026-11-09
norpadon points out a little-known fact: on chips with systolic arrays (Tensor Cores, Neural Accelerators, etc.), verification latency in speculative decoding rises as a staircase as a function of the draft budget, with step widths exactly matching the systolic array width.
Accompanying benchmarks on an M5 Max running Qwen3.8-27B-M show GPU verification time jumping at speculative budgets of 16, 32, and 48 — from 29ms to over 50ms — with the slowdown vs autoregressive spiking from 12% to 66% at budget 32. Practical takeaway: pick speculative decoding draft lengths aligned to array-width boundaries to avoid wasted cycles.
More from Infra
- 92T tokens over 1.54M DAU: the agent-era math feud over token burn — teortaxesTex · 2026-11-09
- Gary Marcus backs Ed Zitron: US data center capacity additions halved in Q4 2025 — GaryMarcus · 2026-11-09
- Analysts: Nvidia could stabilize near $155 if Huang is cleared, but an exit over security violations would hurt far more — GaryMarcus · 2026-11-09
- Texas gubernatorial candidate vows moratorium on all data center approvals — Traditional-Chip8339 · 2026-11-09
- Agents burn tokens far faster than chatbots, warns cloud analyst — DavidLinthicum · 2026-11-09
- Resurfaced report: Nvidia probed Singapore data center firm Megaspeed over China ties — econoar · 2026-11-09