New paper: Tri-Metric Router cuts long-context RAG OOM failures to 0% on a T4
chaumian · x · 2026-09-18
An arXiv paper introduces the Tri-Metric Router, addressing a "Compression Paradox" when deploying RAG on commodity GPUs like the 16GB NVIDIA T4: neural prompt compression adds KV cache contention and preprocessing latency that outweigh savings, while skipping it causes OOM on long contexts.
- Identifies two distinct failure mechanisms when a vLLM-served LLM and a PyTorch compressor co-deploy under tight memory budgets
- A deterministic, training-free policy routes among Raw, Neural (LLMLingua-2), and Lexical (BM25) pipelines
- Dispatch uses three CPU-side signals (spatial complexity, syntactic density, type-token ratio) plus hardware-physical signals — VRAM headroom and a latency crossover point — rather than semantic-only adaptation
- Thresholds are calibrated from profiling LongBench qasper; the crossover sits near 4,332 words on T4, and out-of-distribution holdouts achieve 0% OOM with 88.5±4.4% oracle routing accuracy
Related event: Tri-Metric Router Enables OOM-Free Long-Context RAG on 16GB Consumer GPUs(2 posts)→
More from Infra
- Dev Patches vLLM to Run DiffusionGemma, Live Evals Show It Ties on Smarts but Loses on Speed to APIs — bodonoghue85 · 2026-09-18
- Modular 26.6 Opens Mojo Compiler to External Contributions, Adds Audio Generation in MAX — clattner_llvm · 2026-09-18
- Xiaomi MiMo achieves streaming large-scale LLM RL training, including a 1T-parameter model — stanfordnlp · 2026-09-18
- Google's Gemini managed agents update: 30% lower costs, new Files and Credentials APIs — _philschmid · 2026-09-18
- Top OpenAI researchers reportedly burn $7-8k/day on Codex, growing exponentially — venturetwins · 2026-09-18
- Nebius GB300 NVL72 rack tops MLPerf with 603k tokens/sec on DeepSeek R1 — demian_ai · 2026-09-18