How Quail wins: operator ordering, KV pinned in HBM, and tree attention for AI joins
sh_reya · x · 2026-09-25
Quail's speedups come from jointly optimizing planning and inference: it reorders AI operators and pipelines rows so KV cache stays in GPU HBM, batches work to keep the GPU busy, and uses tree attention tricks (à la SpecInfer and Hydragen) to reuse forward-pass work in AI joins. The author argues AI-SQL is a distinct workload — requests known upfront, throughput-oriented, entirely prefill — justifying a specialized inference engine.
More from Infra
- Modern Microprocessors: A 90-Minute Guide Still the Best Crash Course for Systems Engineers — blaizedsouza · 2026-09-25
- GPU returns hit +76% a year as H100 rents jump 49% and B300 rates climb 66% — KyeGomezB · 2026-09-25
- openjev-sglang: SGLang Radix Cache lets Qwen3.6-35B-A3B run 64 Jev decisions in under 1s — multiply_matrix · 2026-09-25
- Ramp benchmarks Jev to replace LLM reranking: 10x lower tail latency at 300ms, 3x cheaper — multiply_matrix · 2026-09-25
- Qualcomm pitches the phone as the AI hub at Snapdragon Summit, aiming for Apple-like cross-device experience — BenBajarin · 2026-09-25
- Running Qwen3.8-Flash-Next on a 5090 with llama.cpp: 40 tok/s and barely any RAM used — nirurin · 2026-09-25