40k agent calls show prefix stability, not model choice, drives 4-5x cost gaps
MichelleZhao91 · reddit · 2026-11-11
An engineer on the model-access side of a small API gateway shared aggregate billing data from 40,600 agent calls on their own platform (disclosed as self-interested, not a survey).
Key numbers
- 4.04B tokens billed, of which 3.35B were cache reads — 83% of all input landed at the cache rate instead of full input rate.
- Extreme case: one internal agent key runs 397k input tokens per call with 395k cache hits (99.5%) — each turn re-reads the same system prompt and tool schemas, not actual conversation.
Why it beats model choice
With a stable prefix, most input bills at the cache rate: on DeepSeek V4.1 Flash that's $0.0059/M vs $0.2046/M for fresh input, roughly 35x. Two teams on the same model with the same workload can land 4-5x apart on cost without either doing anything wrong.
What breaks cache hits
- Volatile content near the top of the prompt (timestamps, session IDs, tool results landing before the stable block)
- Changing mounted tools between turns
- Switching models mid-loop — cache is scoped per model, so escalation paths re-read everything at full rate
- Idle gaps expiring the provider's prefix
More from coding & agent
- Agent platforms answer 'Can I build it without code?' but almost none answer 'Will it hold in production?' — Ayanle_O · 2026-11-11
- Rork's built-in Mobbin search surfaces 600K+ real app screens as design references, free plan included — rudrank · 2026-11-11
- swyx asks AEO CEO why his agents keep choosing Resend on their own — AccBalanced · 2026-11-11
- Matt Shumer's new video clarifies when to use Fable 5.2 vs Opus 5.5 — prasenx · 2026-11-11
- Rabbit insider: OS3 auto-responds to bug reports and drafts fixes in seconds — jesselyu · 2026-11-11
- Anthropic launches Claude Meetings: Claude joins Google Meet calls and does tasks live — No-Gazelle1365 · 2026-11-11