New paper: Tri-Metric Router cuts long-context RAG OOM failures to 0% on a T4
chaumian · x · 2026-09-18
An arXiv paper introduces the Tri-Metric Router, addressing a "Compression Paradox" when deploying RAG on commodity GPUs like the 16GB NVIDIA T4: neural prompt compression adds KV cache contention and preprocessing latency that outweigh savings, while skipping it causes OOM on long contexts.
- Identifies two distinct failure mechanisms when a vLLM-served LLM and a PyTorch compressor co-deploy under tight memory budgets
- A deterministic, training-free policy routes among Raw, Neural (LLMLingua-2), and Lexical (BM25) pipelines
- Dispatch uses three CPU-side signals (spatial complexity, syntactic density, type-token ratio) plus hardware-physical signals — VRAM headroom and a latency crossover point — rather than semantic-only adaptation
- Thresholds are calibrated from profiling LongBench qasper; the crossover sits near 4,332 words on T4, and out-of-distribution holdouts achieve 0% OOM with 88.5±4.4% oracle routing accuracy
Related event: Tri-Metric Router Enables OOM-Free Long-Context RAG on 16GB Consumer GPUs(2 posts)→
More from Infra
- To dethrone Postgres, new databases must prioritize ecosystem integrations from day one — mgill25 · 2026-11-02
- ESP32-powered 4-inch touchscreen builds a TUI dashboard for monitoring a DGX inference cluster — CATLLM · 2026-11-02
- OpenRouter crosses 10 quadrillion annualized token run rate, doubling every 10.5 weeks — deedydas · 2026-11-02
- Palmer Luckey asks: can you use datacenter RAM for things prohibited by the operator's terms of service? — avaitopiper · 2026-11-02
- Disable MTP for 20-30% faster local LLM inference on limited RAM — InternalMode8159 · 2026-11-02
- Kioxia up 216% in a year but down 68% from its June 2026 peak — zephyr_z9 · 2026-11-02