UC Berkeley and Modal Open-Source Quail, an AI-SQL Inference Engine

Shreya Shankar of UC Berkeley and Charles Frye of Modal released Quail, an open-source (MIT) AI-SQL inference engine for AI-driven data operators: users can write natural-language predicates directly in SQL, e.g. SELECT FROM reviews WHERE AI.IF("the review discusses the ending"), triggering millions of LLM calls. The team reports 1B+ input tokens/min on a single H100 for a single query.

Confirmed

Existing AI-SQL systems only optimize logical and physical plans and ignore the inference stack; Quail co-orchestrates query planning and LLM inference, treating AI operators as first-class, SQL-like operators scheduled as a new inference workload.
Speedups come from joint optimization: reordering AI operators, pipelining row processing, keeping KV cache resident in GPU HBM with batching to saturate the GPU; AI join optimization borrows from speculative execution; tree attention reuses forward computation.
Implementation reuses vLLM internals as a library but replaces the general serving scheduler with a custom one designed for batched, high-throughput, prefill-only AI filter/join queries.
Benchmarks: across 29 queries Quail averages 1.84x faster than a hand-tuned vLLM baseline; a medical-report query is 14x faster (29 min vs 6.84 h), with much lower KV regret.
On the vLLM baseline, many queries hit heavy host-side overhead and vLLM discards KV cache still needed later, causing 50M extra tokens processed; the team estimates a theoretical floor of 15 minutes.
The author acknowledges Quail lacks automatic prefix caching: on agent-trace analysis with shared prefixes, vLLM is 2.32x faster via automatic prefix cache matching; prefix caching is on Quail's roadmap.

Why it matters

AI-SQL brings natural-language predicates into database queries, a new direction combining LLM inference and data systems. Quail shows that jointly optimizing query planning and inference—rather than treating the LLM as a black box—yields order-of-magnitude throughput differences, providing an open reference implementation for high-throughput AI data operators. The author candidly notes it still trails vLLM's automatic prefix caching on repeated-prefix workloads, so its advantages have clear boundaries.

2026-09-25 ~ 2026-09-27 · 25 related posts

Primary sources

Quail: open-source AI-SQL engine hits 1B+ tokens/min on a single H100 — sh_reya · 2026-09-25 03:37 (UTC+08:00)
How Quail wins: operator ordering, KV pinned in HBM, and tree attention for AI joins — sh_reya · 2026-09-25 03:37 (UTC+08:00)
Quail benchmarks: 1.84x faster than hand-tuned vLLM, 14x on medical reports query — sh_reya · 2026-09-25 03:37 (UTC+08:00)

[source] Quail: open-source AI-SQL engine hits 1B+ tokens/min on a single H100 — sh_reya · 2026-09-25
vLLM baseline took 6.84h for an AI-SQL query estimated at 15min speed-of-light — sh_reya · 2026-09-25
[source] How Quail wins: operator ordering, KV pinned in HBM, and tree attention for AI joins — sh_reya · 2026-09-25
[source] Quail benchmarks: 1.84x faster than hand-tuned vLLM, 14x on medical reports query — sh_reya · 2026-09-25
Quail author concedes vLLM's automatic prefix caching wins agent-trace workloads, 2.32x faster — sh_reya · 2026-09-25
Berkeley open-sources Quail, an ultra-high throughput AI-SQL engine with query plans — sh_reya · 2026-09-25
Hitting a billion tokens per minute on one GPU with a query planner plus inference engine — charles_irl · 2026-09-25
Inside Quail: custom vLLM scheduler, workload-aware KV cache for 1B tok/min — sh_reya · 2026-09-25
Quail Debuts: A Domain-Specific LLM Inference Engine for AI-SQL — ShadajL · 2026-09-25

16 near-duplicate retellings: sh_reya · sh_reya · sh_reya · sh_reya · sh_reya · sh_reya · sh_reya · sh_reya · charles_irl · sh_reya · sh_reya · sh_reya · sh_reya · sh_reya · sh_reya · sh_reya