Drafting from the document makes AI extraction 4.7x faster via adaptive speculative decoding
sh_reya · x · 2026-11-10
Shreya Shankar and Charles Frye detail Quail's AI.EXTRACT implementation, which uses speculative decoding with draft tokens copied from the source document, with an adaptive draft length learned during inference.
- 4.7x faster on average (up to 19x) vs a naive vLLM baseline; 1.8x faster (up to 3.0x) vs a hand-tuned vLLM baseline
- Cost per query measured on an H100 at $3.95/hour across four QUAIL-B queries; CUAD-16 (contracts, 41 fields) drops from 2,078.7s to 108.3s
- Charts show draft lengths self-calibrate: converging to 10 tokens for short-answer tasks and 50 for long-answer tasks
More from Infra
- Polymarket odds: 23% chance Anthropic announces bankruptcy by end of 2029 — Polymarket · 2026-11-10
- Benchmark: Agentgateway costs ~$280/yr at 5000 routes, rivals up to 182× more — bibryam · 2026-11-10
- Anthropic signs $3.4B deal to power AI data centers with natural gas generators, bypassing the grid — Polymarket · 2026-11-10
- NVIDIA's Jacob Loewenstein: token efficiency matters more than price sheets when picking models — togethercompute · 2026-11-10
- AI networking market to hit $278.9B by 2027 at 95% CAGR, TrendForce forecasts — Beth_Kindig · 2026-11-10
- Peter Diamandis: Data center backlash is the best thing to happen to AI — PeterDiamandis · 2026-11-10