How one AI agent cut serving costs 37% on 1M weekly messages, in 13 production changes
AccBalanced · x · 2026-11-04
An AI agent developer shares how they cut serving costs by 37% on 1M weekly messages without changing prices, with 13 concrete production changes and numbers:
- Track cache reads, not input/output tokens: on GPT-6 Sol, cached input costs $0.20/M vs $2 fresh; their chat now reads 84–90% of every prompt from cache — the single biggest bill mover.
- Split the system prompt: static parts (instructions, rules, examples) first and cached; anything workspace- or turn-specific goes in a second block so it never breaks the cache above.
- Keep system prompt and tools byte-identical between turns: OpenAI caches tools and system messages as one block — one changed character means 0 cached tokens.
Quoted context: Bloomberg reports Harvey's gross margin fell from 50% to -50% by June as agent token usage spiked 20x on rented OpenAI/Anthropic models — underscoring why inference cost engineering matters.
More from coding & agent
- pr-agent, the original open-source AI PR reviewer, hits 14.5k stars and says it's not the Qodo free tier — The-PR-Agent · 2026-11-04
- NVIDIA open-sources auto-ontology, a Python framework for graph-based reasoning over LLM and agent workflows — NVIDIA · 2026-11-04
- The Best Claude Setup Is No Setup: Auto Mode + Let It Write Code — rubenhassid · 2026-11-04
- Indie Dev Built 10 Apps in 10 Months, Now Makes $800K/yr: Full Playbook — tinyfool · 2026-11-04
- GitHub Copilot lands in JetBrains Air as an agent, locally and in cloud sandboxes — mariorod1 · 2026-11-04
- Claude Code system prompt diff: stricter email privacy, safer deletes, clearer game rules — rchardkovacs · 2026-11-04