ByteDance Seed's HarnessDev scores agents on harnesses they build, not tasks they finish
omarsar0 · x · 2026-09-02
ByteDance Seed's HarnessDev paper proposes scoring self-evolving agents on the execution harness they build rather than completed tasks.
Method:
- Agent starts from a weak runnable seed plus a few cases and builds a full execution system
- A second stage iterates the harness on downstream feedback
- Both stages scored on capability and execution-token cost
Experiments: 6 creator LLMs, 4 domains, 2,207 held-out instances.
Findings: generated harnesses trail mature human engineering on code and research, but match or beat it on writing and ML experimentation — a sharp domain split.
More from coding & agent
- OpenAI gpt-live-1 tip: control voice sessions from your backend via sideband connections — pbbakkum · 2026-10-21
- Developer builds himself a 'Rork MoTS' agent on Grok Bot, aiming for full autonomy before GTA 6 — rudrank · 2026-10-21
- OpenAI Live API tip: control voice sessions from your backend via sideband WebSocket — pbbakkum · 2026-10-21
- OpenAI's MultiLangBench: PHP scores highest (91.3%) at building a database engine from scratch — ricklamers · 2026-10-21
- UserProof lets Claude Code watch your user session recordings and boost conversion for $29/mo — doooyle · 2026-10-21
- dphnAI's distributed inference network served 32B DeepSeek tokens in 24h, ~6% of Venice's daily volume — Ronangmi · 2026-10-21