NVIDIA details agentic inference economics: Claude Code hits 85-97% cache, 11.7x read/write ratio
_ScottCondron · x · 2026-08-23
NVIDIA's technical blog post "Full-Stack Optimizations for Agentic Inference with Dynamo" explains how to bring managed-API-grade agent inference optimization to self-hosted open-source models.
Why it matters: Coding agents now write production code at scale — Stripe's agents generate 1,300+ PRs per week, Ramp attributes 30% of merged PRs to agents, Spotify reports 650+/month. Claude Code makes hundreds of API calls per session, each carrying the full history, putting heavy pressure on KV cache.
Key numbers: After the first call writes the conversation prefix, subsequent calls hit 85-97% cache; a 4-Opus agent swarm reaches 97.2% aggregate hit rate with an 11.7x read/write ratio — a write-once-read-many pattern. Maximizing cross-worker cache reuse and keeping KV blocks warm and routable is the central optimization target.
Dynamo's approach: Making Dynamo agent-native at three layers — frontend API, router, and KV cache management — so self-run GPU clusters get prefix matching, cache placement, and eviction that managed providers control out of the box.
Related event: NVIDIA details agent-aware inference caching with up to 97% hit rate(2 posts)→
More from coding & agent
- Coarena Launches Crowdsourced Benchmark to Fix Computer-Use Data Leakage — ycombinator · 2026-08-23
- AdaL launches free Loop Engineering course: build AI SaaS with agents — Zachly · 2026-08-23
- How Exa saves Agent compute by skipping browser DOM parsing — yoimnotkesku · 2026-08-23
- App Store Connect CLI 4.8.0 Released with Xcode Cloud Doctor Feature — rudrank · 2026-08-23
- AI writes almost all code, engineers shift to high-level design — deanwball · 2026-08-23
- $10k AI vulnerability scan completed in 10 hours — scaling01 · 2026-08-23