Hitting a billion tokens per minute on one GPU with a query planner plus inference engine
charles_irl · x · 2026-09-25
Modal's Charles Frye and CMU's Shreya Shankar detail how they hit a billion tokens per minute on a single GPU by combining a database-style query planner with an LLM inference engine, targeting AI-SQL-style bulk inference workloads.
The piece argues a quieter inference revolution is happening behind coding agents and chatbots: simple LLM transformations of massive data don't need frontier intelligence, but naively feeding millions of multi-thousand-token sequences into low-latency-optimized engines wastes huge amounts of compute. Planning queries like a database and batch-scheduling them to the engine cuts cost dramatically — a point on the Pareto curve of latent demand that the race for higher intelligence overlooked. Karpathy also weighed in.
Related event: UC Berkeley and Modal Open-Source Quail, an AI-SQL Inference Engine(25 posts)→
More from Infra
- Gas-for-AI-datacenters forum co-located with CoRL shows energy demand spillover — carlosdponx · 2026-11-12
- a16z podcast: Mike Volpi on Cisco's 70+ acquisition playbook and the AI buildout — a16z · 2026-11-12
- AMD to pass Intel in server revenue by mid-2027 as share gains 6 points a year — BenBajarin · 2026-11-12
- Study: Reflective Data Center Roofs Actually Cool Daytime Ground by 1.8-2.2°C — SumitGup · 2026-11-12
- Tsinghua's BaGuaLu-2 tames MoE expert-load spikes at runtime, accepted to SC26 — jiqizhixin · 2026-11-12
- antirez backs low-bit DeepSeek 4.1 Flash over Qwen 27B dense on Apple M5 Macs — antirez · 2026-11-12