Speculative decoding has been built into HF transformers forever: just pass assistant_model= in generate()
cephaloform · x · 2026-11-18
Handy reminder: speculative decoding has long been built into Hugging Face transformers. No extra library needed — just pass assistantmodel= pointing to your drafter model in model.generate(), and a smaller draft model proposes tokens for the big model to verify, speeding up generation.
More from Infra
- Decentralized inference platform Chutes partners with Alibaba Cloud — bittingthembits · 2026-11-18
- Nvidia raises FY2028 outlook to ~75% revenue growth, up from 70%, off a higher base — firstadopter · 2026-11-18
- Nvidia breaks 8-quarter streak: no 10-Q filed alongside earnings release — churchkey · 2026-11-18
- TensorScale visualizes its generation pipeline and publishes head-to-head inference benchmarks vs a leading host — TerryTangYuan · 2026-11-18
- SayGM lists GPT-6.2 Astra 4 hours after launch, 12.4% under OpenAI list price — markjeffrey · 2026-11-18
- SpaceX targets first AI computing satellites in 2027, gigawatt-scale orbital compute by 2029 — Polymarket · 2026-11-18