Unify Cuts AI Agent Costs 95% by Bypassing OpenAI Cache Limits
LangChain · x · 2026-09-01
Unify CTO Connor Heggie details the engineering journey to slash AI agent costs by 90-95%. The key insight involved routing around OpenAI's 15 requests-per-second limit on prompt caching. By building a custom routing layer and treating subagents as simple function calls rather than distinct entities, they achieved a 95% cache hit rate. The discussion also covers why LLM judges must belong to a different model family, the benefits of auditing speed sequentially rather than in bulk, and the shift from full VMs to a Python REPL for running durable cloud agents.
Related event: Self-Built Routing Bypasses OpenAI Cache Limits to Cut AI Costs 95%(2 posts)→
More from coding & agent
- Matt Pocock's pre-commit hook caps agent-written files at 5,000 tokens — mattpocockuk · 2026-10-18
- Author Laurence Moroney: write your ugly first draft yourself, then let an LLM edit it — lmoroney · 2026-10-18
- Open-weight TYGER speech decision model makes 80ms turn-taking calls for full-duplex voice agents — solyarisoftware · 2026-10-18
- Devs: Opus 5.5 on medium beats Opus on max — eyishazyer · 2026-10-18
- He Stopped Telling Claude Code to Stop Making Up Timestamps—and Made It Impossible with a PreToolUse Hook — Grimmoner · 2026-10-18
- Cursor Super + Claude Pro + OpenCode: can this stack survive heavy autonomous coding? — DarioAmodeiWasTaken · 2026-10-18