Deep Inference-Query Engine Integration: Custom Scheduler and Workload-Aware KV Cache for Prefill-Only AI Filters
charles_irl · x · 2026-09-25
A deep engineering discussion on integrating inference engines with query engines: instead of treating the LLM as a generic serving endpoint, the team uses vLLM internals as a library and replaces its general-purpose scheduler with one designed for batched, high-throughput, prefill-only queries used for AI filters/joins.
Key points:
- The custom scheduler enables workload-aware KV cache management
- A custom execution layer implements attention kernels that avoid re-reading the same shared KV across multiple queries
- shreya was surprised by the headroom on prefill-only queries: in SQL the query shape (e.g. join structure) is known in advance, so it makes sense to implement attention differently; expects the same for other operators like semantic groupby
- Quote context likens this to workload-aware inference schedulers for agent workloads, which use session-aware KV TTLs to avoid cache eviction during environment interactions
More from Infra
- Investor Predicts EDA/CAD Will Collapse Into One Flow Within 3-5 Years — ai · 2026-09-25
- AI Data Center Debt Starting to Roll Over, Rising Rates Accelerating the Problem — AIFlow_ML · 2026-09-25
- Musk details xAI compute: Colossus 2 to hit 880k GB300s by year-end — elonmusk · 2026-09-25
- Qwen-Image-2.1 gets GGUF quantization, could run text-to-image on a Snapdragon 865 phone — ResidentAping · 2026-09-25
- Finance worker seeks local AI setups to cut soaring Codex/ChatGPT costs — Startup__Sam · 2026-09-25
- Diffusion LLM goes production: Augment Code's Mercury 2.5 switch cuts latency 82%, cost 90% — cen6wkf · 2026-09-25