Harness design drives AI inference energy: cache misses can double a request's energy use
bibryam · x · 2026-10-25
Neuralwatt analyzed a month of production traffic and found the agent harness has an outsized impact on inference energy:
- Cache hit rates vary widely by client: deepseek-harness leaves only 1.6% of input tokens uncached, while Node.js apps hit 11.1%. At 92% cached vs 97%, prefill work differs by 2.7x.
- A cache miss roughly doubles a request's energy — about 2.5x on GLM-5.3.
- Deep sessions amplify the cost: past 128k tokens, one turn re-reading more than half its prompt costs as much prefill as 600 normal turns.
- It's fixable: every cache break observed came from a harness editing content it had already sent, forcing a re-read from the first difference.
The author argues inference efficiency discussions usually focus on models and GPUs, but the client harness deserves equal scrutiny.
More from coding & agent
- 10 open-source GitHub repos that put every AI model behind one API — victor_explore · 2026-10-25
- Viajante: Open-Source MCP Server Scrapes Real Flight and Hotel Prices, No API Keys — Fast_Entrance7839 · 2026-10-25
- HCI researcher bets AI can finally make Bayesian stats usable for small-N studies — IanArawjo · 2026-10-25
- iOS demo combines GPT-Live-1, Decisions API and Stream Video for real-time vision — amos_gyamfi · 2026-10-25
- Redditor builds R-rated comedy club where AI agents do stand-up — and yours can sign up — bshane54 · 2026-10-25
- 30 Days, 227 Commits: Developer Builds a Game Solo with Claude Opus — xiaohu · 2026-10-25