How Quail wins: operator ordering, KV pinned in HBM, and tree attention for AI joins
sh_reya · x · 2026-09-25
Quail's speedups come from jointly optimizing planning and inference: it reorders AI operators and pipelines rows so KV cache stays in GPU HBM, batches work to keep the GPU busy, and uses tree attention tricks (à la SpecInfer and Hydragen) to reuse forward-pass work in AI joins. The author argues AI-SQL is a distinct workload — requests known upfront, throughput-oriented, entirely prefill — justifying a specialized inference engine.
Related event: UC Berkeley and Modal Open-Source Quail, an AI-SQL Inference Engine(25 posts)→
More from Infra
- Gas-for-AI-datacenters forum co-located with CoRL shows energy demand spillover — carlosdponx · 2026-11-12
- a16z podcast: Mike Volpi on Cisco's 70+ acquisition playbook and the AI buildout — a16z · 2026-11-12
- AMD to pass Intel in server revenue by mid-2027 as share gains 6 points a year — BenBajarin · 2026-11-12
- Study: Reflective Data Center Roofs Actually Cool Daytime Ground by 1.8-2.2°C — SumitGup · 2026-11-12
- Tsinghua's BaGuaLu-2 tames MoE expert-load spikes at runtime, accepted to SC26 — jiqizhixin · 2026-11-12
- antirez backs low-bit DeepSeek 4.1 Flash over Qwen 27B dense on Apple M5 Macs — antirez · 2026-11-12