Walmart's Training-Free Tri-Metric Router Cuts RAG OOM Failures to 0% on 16GB GPUs
_reachsumit · x · 2026-09-17
A Walmart arXiv paper identifies the "Compression Paradox" when deploying RAG on commodity GPUs (e.g., NVIDIA T4, 16GB VRAM): neural prompt compression adds KV-cache contention and preprocessing latency that outweigh generation savings, while skipping compression causes OOM on long contexts.
They propose the Tri-Metric Router, a deterministic, training-free policy choosing among Raw, Neural (LLMLingua-2), and Lexical (BM25) pipelines:
- Dispatch signals are hardware-physical — VRAM headroom and a latency crossover point — plus three CPU-side metrics: spatial complexity, syntactic density, and type-token ratio.
- Thresholds are calibrated via profiling on LongBench qasper, yielding a crossover near 4,332 words on T4; the contribution is the calibration methodology, not a hardware constant.
- On out-of-distribution holdouts: 0% OOM failures and 88.5±4.4% of oracle performance.
More from Infra
- GLM agent built its own inference infra in two weeks, tripling end-to-end throughput — jietang · 2026-09-17
- Open-source PolyServe autotuner boosts LLM throughput 56–97% across three GPUs — AffectionateSir8341 · 2026-09-17
- China Telecom open-sources Xing4.0-29B MoE, first in class trained fully on Ascend NPUs — Skyline34rGt · 2026-09-17
- Leaked PR Hints Circuit & Chisel's GPU Router Adds "union-alpha" Per-Account Name for Pareto — cephaloform · 2026-09-17
- 34GB Model Stack on an 8GB RTX 4060: LTX2.5 Generates 10s Video Locally — BigBullshitta · 2026-09-17
- Nebius raises on-demand GPU prices up to 21% starting Oct 1 — sudoraohacker · 2026-09-17