UC Berkeley Open-Sources Quail, an AI-SQL Engine Hitting 1B+ Tokens/Min on a Single H100
UC Berkeley's Full-Stack Data Lab, together with Charles Frye of Modal, has released Quail, an open-source AI-SQL inference engine under the MIT license, along with a blog post, code, and an online demo. It targets an emerging use case—"AI-powered data operators": users can write natural-language conditions directly in SQL, e.g. SELECT FROM reviews WHERE AI.IF("the review discusses the ending"), which can trigger millions of LLM calls. The team reports a throughput of 1B+ input tokens/min for a single query on a single H100.
Confirmed
- Quail is open-sourced under the MIT license by a UC Berkeley team, in collaboration with Charles Frye of Modal, with a blog post, code, and an online demo available.
- The speedup comes from jointly optimizing query planning and inference: AI operators are reordered and row processing is pipelined so the KV cache stays resident in GPU HBM, with batching keeping the GPU fully utilized; AI join optimization borrows ideas from Speculative Execution; and tree attention is used to reuse forward computation.
- The team disclosed comparison data: running all LLM calls of AI-SQL queries on a hand-tuned vLLM baseline, multiple queries showed heavy host-side overhead, and vLLM discarded KV caches that were still needed, resulting in roughly 50 million extra tokens processed; the vLLM baseline took 6.84 hours, while the team estimates the theoretical limit at about 15 minutes.
Why it matters
- AI-SQL brings natural-language conditions directly into database queries, a new direction combining LLM inference with data systems; Quail shows that jointly optimizing query planning and inference (rather than treating the LLM as a black box) can yield order-of-magnitude throughput differences, providing an open-source reference implementation for high-throughput AI data operators.
2026-09-25 ~ 2026-09-25 · 9 related posts
Primary sources
- [source] Quail: open-source AI-SQL engine hits 1B+ tokens/min on a single H100 — sh_reya · 2026-09-25
- vLLM baseline took 6.84h for an AI-SQL query estimated at 15min speed-of-light — sh_reya · 2026-09-25
- How Quail wins: operator ordering, KV pinned in HBM, and tree attention for AI joins — sh_reya · 2026-09-25
- [source] Quail benchmarks: 1.84x faster than hand-tuned vLLM, 14x on medical reports query — sh_reya · 2026-09-25
- Quail author concedes vLLM's automatic prefix caching wins agent-trace workloads, 2.32x faster — sh_reya · 2026-09-25
- [source] Berkeley open-sources Quail, an ultra-high throughput AI-SQL engine with query plans — sh_reya · 2026-09-25