Why vLLM takes 6.84 hours where speed-of-light is 15 minutes on AI-SQL queries
sh_reya · x · 2026-09-25
Running AI-SQL through hand-tuned vLLM baselines exposed two failures: heavy host overhead from many LLM calls, and KV cache eviction that forced reprocessing of 50 million extra tokens — a 6.84-hour query versus a 15-minute speed-of-light estimate. The author argues AI-SQL is a distinct workload (requests known upfront, KV plannable, throughput-oriented, entirely prefill), motivating the specialized inference engine Quail.
More from Infra
- Anthropic paid $5B for short-term compute in Q3 as spot prices run ~2x long-term deals — tengyanAI · 2026-11-11
- UK Sovereign AI invests in Fractile to build next-gen frontier AI chips — charlieharris01 · 2026-11-11
- What happened when I moved vector retrieval into Snowflake SQL instead of orchestrating RAG outside — dima806_dima · 2026-11-11
- AI chip startup Fractile raises $720M Series C for DRAM-on-processor design — Tom_Westgarth15 · 2026-11-11
- Mobile AI Factory: agentic pipeline builds on-device SLMs that beat 5-shot frontier models — kalyan_kpl · 2026-11-11
- Vivo's Starlink direct-to-cell service hits 10M users in two weeks across Brazil — XFreeze · 2026-11-11