Speculative decoding has been built into HF transformers forever: just pass assistant_model= in generate()

cephaloform · x · 2026-11-18

Handy reminder: speculative decoding has long been built into Hugging Face transformers. No extra library needed — just pass assistantmodel= pointing to your drafter model in model.generate(), and a smaller draft model proposes tokens for the big model to verify, speeding up generation.

Original post →

More from Infra

Infra channel →