Why vLLM takes 6.84 hours where speed-of-light is 15 minutes on AI-SQL queries
sh_reya · x · 2026-09-25
Running AI-SQL through hand-tuned vLLM baselines exposed two failures: heavy host overhead from many LLM calls, and KV cache eviction that forced reprocessing of 50 million extra tokens — a 6.84-hour query versus a 15-minute speed-of-light estimate. The author argues AI-SQL is a distinct workload (requests known upfront, KV plannable, throughput-oriented, entirely prefill), motivating the specialized inference engine Quail.
More from Infra
- Investor Predicts EDA/CAD Will Collapse Into One Flow Within 3-5 Years — ai · 2026-09-25
- AI Data Center Debt Starting to Roll Over, Rising Rates Accelerating the Problem — AIFlow_ML · 2026-09-25
- Musk details xAI compute: Colossus 2 to hit 880k GB300s by year-end — elonmusk · 2026-09-25
- Qwen-Image-2.1 gets GGUF quantization, could run text-to-image on a Snapdragon 865 phone — ResidentAping · 2026-09-25
- Deep Inference-Query Engine Integration: Custom Scheduler and Workload-Aware KV Cache for Prefill-Only AI Filters — charles_irl · 2026-09-25
- Finance worker seeks local AI setups to cut soaring Codex/ChatGPT costs — Startup__Sam · 2026-09-25