Inside Quail: custom vLLM scheduler, workload-aware KV cache for 1B tok/min
sh_reya · x · 2026-09-25
- A technical breakdown of Quail: the team used vLLM internals as a library but swapped the general serving scheduler for one designed for batched, high-throughput, prefill-only AI filter/join queries.
- The custom scheduler adds workload-aware KV cache management, and attention kernels avoid re-reading shared KVs across queries—key to hitting 1B+ input tokens/min on one H100.
- The author expects more workload-specific inference schedulers as LLM usage gets creative, aligning with the rise of classification-heavy workflows like jev.
Related event: UC Berkeley and Modal Open-Source Quail, an AI-SQL Inference Engine(25 posts)→
More from coding & agent
- Stripe launches claimable sandboxes: frictionless payment integrations for AI agents — jeff_weinstein · 2026-11-12
- Databricks launches autonomous AI coworker Genie: 91% on enterprise tasks with half the tokens — Yuchenj_UW · 2026-11-12
- Loom founder launches Roma 2.0, a shared to-do list that orchestrates AI agents — thetripathi58 · 2026-11-12
- Developer proposes a 'blast radius' test for AI agents before building — Few-Garlic2725 · 2026-11-12
- Sosumi Eval: Apple Docs Access Lifts Coding Agent Pass Rates from ~33% to 94-98% — JordanMorgan10 · 2026-11-12
- CypherGuard fixes structurally wrong LLM Cypher queries, hitting 88.5% on CypherBench — JeremyCMorgan · 2026-11-12