Why autoregressive LLM decoding is slow: the memory bandwidth bottleneck
cneuralnetwork · x · 2026-10-22
Part one of @cneuralnetwork's series on speculative decoding explains why autoregressive decoding is slow:
- Every token depends on previous ones, so generating n tokens requires n sequential forward passes;
- At small batch sizes, inference is largely memory-bandwidth bound — model weights are streamed through memory just to emit a single token, leaving compute idle.
Related event: Illustrated Guide to Speculative Decoding(3 posts)→
More from Research
- Neural networks 'clean and transparent'? Turing Post weighs both sides of the interpretability debate — TheTuringPost · 2026-10-22
- Holon-S1, never trained on osteosarcoma, reproduces clinicians' molecular finding from slides — iScienceLuvr · 2026-10-22
- ATEST: load-aware test-time scaling cuts LLM inference energy up to 48x — Mobina Kashaniyan · 2026-10-22
- New tetrahedron theorem found with Claude after scanning 1.6M sphere combos — IgorCarron · 2026-10-22
- ROME attacks: poisoned Python can surgically rewrite LLM weights in GPU memory — SirJerryLion · 2026-10-22
- UC Berkeley opens joint Statistics×EECS tenure-track hire in Trustworthy AI — jasondeanlee · 2026-10-22