ByteDance's HarnessDev: LLMs can build their own agent harnesses, but they don't transfer
rohanpaul_ai · x · 2026-09-04
A new ByteDance paper, "HarnessDev: Can LLMs Create and Evolve Their Own Agent Harness?", starts creator models from a nearly empty runtime and asks them to build execution loops, tools, context handling, recovery, and verification. Key findings:
- Generated harnesses are useful but very uneven across domains: Opus 4.8 self-evaluates at 84.6 on writing (vs 83.7 for the selected external reference) but only 52.4 on BrowseComp vs 92.2.
- Portability is poor: the same Opus-built harness scores 69.3 on SWE-Pro with Opus executing, but drops to 33.0 when Gemini 3.1 Pro runs it.
Conclusion: harness designs co-adapt to a single model rather than becoming a clean, reusable software layer. Paper: arxiv.org/abs/2609.01437.
More from coding & agent
- Developer builds himself a 'Rork MoTS' agent on Grok Bot, aiming for full autonomy before GTA 6 — rudrank · 2026-10-21
- OpenAI Live API tip: control voice sessions from your backend via sideband WebSocket — pbbakkum · 2026-10-21
- OpenAI's MultiLangBench: PHP scores highest (91.3%) at building a database engine from scratch — ricklamers · 2026-10-21
- UserProof lets Claude Code watch your user session recordings and boost conversion for $29/mo — doooyle · 2026-10-21
- dphnAI's distributed inference network served 32B DeepSeek tokens in 24h, ~6% of Venice's daily volume — Ronangmi · 2026-10-21
- Superconductor revamps agent picker spanning 8 coding agents with search and favorites — sergeykarayev · 2026-10-21