WEKA open-sources kvlab: replay real agent traffic to benchmark KV cache offload
AccBalanced · x · 2026-11-16
WEKA released kvlab, an open-source toolkit that captures real agentic traffic and replays it against inference endpoints to measure what happens when agent context outgrows GPU memory. The guide covers GPU time budgets for agentic work, benchmark methodology (realism levels, capacity method, fixed-pool mode), result interpretation with 3D views, and measured outcomes: on Nebius B300 and GLM-5.3 capacity runs, KV offload lifted throughput from 94 out/s (HBM only) to 729 out/s (HBM+DRAM) and 848 out/s (HBM+WEKA AMG). Includes a tool to size your own dataset and see if KV offload helps your traffic.
More from coding & agent
- Three classes of programming languages in the AI era: natural language, performant code, and specs — dbreunig · 2026-11-16
- Turning newsletter edits into AI checks lifts frontier models 25+ points on summaries — danshipper · 2026-11-16
- levelsio runs every business on VPS servers managed solely via SSH and Claude Code — alvelda · 2026-11-16
- Dev builds remote MCP server with inline MCP Apps video player for 5,500 meme clips — Mediocre-Number-5498 · 2026-11-16
- Developer's autonomous "factory" builds usable app for ~$40 in Claude credits — threepointone · 2026-11-16
- 20-year AI veteran curates 19 free open-source repos for building AI agents — MaryamMiradi · 2026-11-16