Walmart's Training-Free Tri-Metric Router Cuts RAG OOM Failures to 0% on 16GB GPUs
_reachsumit · x · 2026-09-17
A Walmart arXiv paper identifies the "Compression Paradox" when deploying RAG on commodity GPUs (e.g., NVIDIA T4, 16GB VRAM): neural prompt compression adds KV-cache contention and preprocessing latency that outweigh generation savings, while skipping compression causes OOM on long contexts.
They propose the Tri-Metric Router, a deterministic, training-free policy choosing among Raw, Neural (LLMLingua-2), and Lexical (BM25) pipelines:
- Dispatch signals are hardware-physical — VRAM headroom and a latency crossover point — plus three CPU-side metrics: spatial complexity, syntactic density, and type-token ratio.
- Thresholds are calibrated via profiling on LongBench qasper, yielding a crossover near 4,332 words on T4; the contribution is the calibration methodology, not a hardware constant.
- On out-of-distribution holdouts: 0% OOM failures and 88.5±4.4% of oracle performance.
Related event: Tri-Metric Router Enables OOM-Free Long-Context RAG on 16GB Consumer GPUs(2 posts)→
More from Infra
- Big Tech's Future Data Center Leases Top $1.3T, 70% Added in Last Four Quarters — rohanpaul_ai · 2026-11-02
- Daniel Lemire: AI is exploding demand for boring compute, server CPUs to double — lemire · 2026-11-02
- Physical AI's real story: a 4-layer map of 15 companies behind humanoid robots — pdamodaran · 2026-11-02
- Under 5% of US consumers pay for AI, yet compute demand is already forcing usage cuts — NinaDSchick · 2026-11-02
- Sam Altman: AI query energy cost fell 1,000x-100,000x in 3-4 years — rohanpaul_ai · 2026-11-02
- Marvell and Lumentum to merge, targeting $550B AI connectivity market — BenBajarin · 2026-11-02