SpecPilot: Berkeley and NVIDIA's Agent Loop Auto-Optimizes Speculative Decoding per Deployment
UCBerkeley · hf · 2026-11-04
UC Berkeley, Princeton, Stanford, and NVIDIA propose SpecPilot, an autonomous agentic loop that discovers deployment-specific speculative decoding algorithms and system optimizations.
Three challenges addressed:
- With a single request, each step accepts few tokens yet reads all target weights
- With batching, per-request verification work roughly halves the speedup over autoregressive decoding
- Speedups are deployment-specific (serving regime, drafter, target model, GPU) with no universal optimum
Each round, a cost model locates the largest gaps to the ceiling; parallel agents explore directions and integrate successful ones, with token forcing checking every output token against the stock system to guarantee no quality loss. Code is open-sourced.
More from Infra
- Stanford's ParallelKernelBench lands NeurIPS 2026: LLMs nail only 32% of multi-GPU kernels — simonguozirui · 2026-11-04
- AMD exec joins Fireworks AI Forge event to highlight multigenerational partnership — AnushElangovan · 2026-11-04
- Microsoft details Outlook search overhaul: 7 relevance-ranked results and offline resilience — benjamingauthey · 2026-11-04
- AMD exec gifts DHH a 192GB AMD workstation, embraces Omarchy Linux — AnushElangovan · 2026-11-04
- How LLM inference actually works: from 140GB weights to GPUs and model serving, explained as a story — _jaydeepkarale · 2026-11-04
- Running 4x RTX 3090 on a consumer board: a Qwen3.8 local inference build — Civil_Fee_7862 · 2026-11-04