Nebius/WEKA benchmark: shared KV cache lifts agentic inference throughput 2.4x with 93% hit rate

AccBalanced · x · 2026-09-25

Nebius and WEKA ran an 8-hour soak test of WEKA NeuralMesh's Augmented Memory Grid (distributed KV cache) on NVIDIA HGX B300, replaying real agentic coding traffic against DeepSeek-V4-Pro.

• Result: 2.4x requests served on the same node, with a 93% cache hit rate vs 40% on GPU HBM alone.
• Why it matters: every agent tool call is a new inference request carrying the full session history; once KV blocks are evicted, the next turn prefills everything from scratch, and host-DRAM spills can't serve other nodes.
• Method: only the cache tier changed — same 8-GPU B300 node at TP=8, same vLLM stack, 512 active sessions drawn from 5,000+ captured agentic sessions.
• The author argues lab-favored vendor benchmarks are bunk and calls for real workloads, open traces, and long-horizon agents tested directly by top AI clouds.

Original post →

More from Infra

• Open-source AI-SQL engine Quail hits 1B+ input tokens/min on a single H100 — sh_reya · 2026-09-25
• vLLM baseline took 6.84h for an AI-SQL query estimated at 15min speed-of-light — sh_reya · 2026-09-25
• How GPUs really run deep learning: a primer on memory hierarchy and optimization — goyal__pramod · 2026-09-25
• VeriTile embeds Triton GPU kernels in Lean, with AI agents writing machine-checked correctness proofs — KaiyuYang4 · 2026-09-25
• Merge Gateway Launches Batch Inference at ~50% of Standard Prices — shensi · 2026-09-25
• New deep-dive article on scaling LLM inference in production — abhijithneil · 2026-09-25

Infra channel →