Quail: open-source AI-SQL engine hits 1B+ tokens/min on a single H100
sh_reya · x · 2026-09-27
- Shreya Shankar (CMU) and Charles Frye (Modal) released Quail, an open-source AI-SQL engine built in collaboration with Modal.
- By planning SQL queries and LLM inference together, it reaches 1B+ input tokens/min on a single H100 for one query, claiming >10x speedup over vLLM.
- AI-SQL embeds AI operators in SQL (e.g. AI.IF(PROMPT(...))) to run prompts over entire tables — a workload producing millions of multi-thousand-token sequences that needs little frontier intelligence, so small open-weights models suffice. This "no thinking, single token, low latency" regime is an under-invested point on the LLM pareto curve.
- The 15-minute Modal blog post covers why AI-SQL matters, how the speedup works, and the future of inference engineering.
Related event: Open-Source AI-SQL Engine Quail Hits 1B Tokens per Minute on One H100(2 posts)→
More from Infra
- Columbia report: 182.7 GW of US data centers must yield $3.7T annual revenue by 2032 — rohanpaul_ai · 2026-09-27
- KoboldCpp v1.122 released for local LLM inference — Fcking_Chuck · 2026-09-27
- Tiny classifiers need no hosting — you can even train them on a phone — max_paperclips · 2026-09-27
- ~9,900 NVIDIA B300 GPUs coming online Dec–Feb, up to 4,096-GPU clusters — sdand · 2026-09-27
- Building a workload-specific inference engine fast by standing on FOSS giants — charles_irl · 2026-09-27
- Australian officials push back on US-imported datacentre backlash: 'We are not the United States' — nordicinst · 2026-09-27