Same model, half the tokens: benchmark shows the agent harness is not neutral plumbing
uriwa · reddit · 2026-11-11
Agent benchmarks almost always compare models while treating the harness as neutral plumbing. To measure the harness cost itself, the author ran pi-coding-agent and his own production messaging harness on the exact same multi-file repo bugs, both using gemini-3.8-flash.
- pi: 36 LLM calls, 248,755 total tokens (8,711 output)
- Own harness: 16 calls, 120,857 total tokens (944 output)
- Both passed the test suites
Two factors explain nearly all the gap:
- Startup tax: 4,200 tokens of default instructions/tool boilerplate vs 800 — the whole prefix is re-read and billed on every turn, compounding over a trajectory.
- Completion bloat: asking for conciseness in the system prompt fails over 25-step trajectories. What works is separating internal tool execution from outward user replies in the event loop schema, and keeping history prefixes deterministic so prompt caching stays above 55%.
Takeaway: the harness is a first-class variable in agent cost and behavior, not plumbing.
Related event: Berkeley Paper: Agent Harness Choice Can Cut Costs 2.6x(2 posts)→
More from coding & agent
- Claude now embeds interactive websites via MCP apps with frame_domains support — round · 2026-11-11
- Collaborator model Ampersand lands on Vercel AI Gateway, free for a week at 744B params — ericzelikman · 2026-11-11
- AI agents already spoofed their own investigation logs in the Hugging Face breach — can tamper-proof audit trails fix it? — 0xSammy · 2026-11-11
- FastMCP team takes over GitHub HQ on Dec 2 for MCP skills and auth talks — LangChain · 2026-11-11
- Mintlify open-sources agent-traffic: agents made 489M doc requests, 77% of all traffic — handotdev · 2026-11-11
- Baseten details deep research agent training: breadth-rewarding RL plus SFT for report writing — baseten · 2026-11-11