Copy-from-document speculative decoding speeds AI extraction 4.7x (up to 19x) vs naive vLLM
sh_reya · x · 2026-11-10
- FS Data Lab's blog details their AI.EXTRACT implementation in Quail: speculative decoding whose draft tokens are copied directly from the input document, with draft length learned adaptively during inference.
- Results: 4.7x average speedup (up to 19x) over a naive vLLM baseline and 1.8x average (up to 3.0x) over a hand-tuned vLLM baseline, with per-query cost dropping on an H100 at $3.95/hr.
- Key insight: extraction answers are usually verbatim spans of the document, so copy-based drafts have high acceptance rates; charts show short-answer queries converge to draft length 10, long-answer ones to 50.
More from Infra
- Google ships four TPU systems in one generation, adds confidential inference and max-context variants — AccBalanced · 2026-11-10
- The math behind Anthropic's 5-minute and 1-hour prompt cache options — bigaiguy · 2026-11-10
- Why Send This to the Cloud? Resume Screening Runs Locally on an RTX Spark — draginol · 2026-11-10
- Signal65 MLPerf analysis: Azure GB300 tops DeepSeek-R1 interactive serving at 260k tokens/sec — ryanshrout · 2026-11-10
- Compute is the first crux of AI takeoff: substitution estimates diverge wildly — austinc3301 · 2026-11-10
- CoreWeave Launches Claude Code Skill to Build GPU Clusters and Serve Models from the CLI — altryne · 2026-11-10