Harness-of-Harness carries state across runs: 71.52 vs 58.24 on long-horizon coding
rohanpaul_ai · x · 2026-09-06
Harness-of-Harness (HoH) tackles the core failure modes of long-horizon coding agents: forgetting earlier decisions, repeating work, breaking working features, and missing unfinished requirements.
- Method: carry the current software, test evidence, known issues, and an updated plan into the next coding run, with independent testing and replanning from real failures.
- Results: improved all 3 tested agent setups across 3 benchmarks. Clearest comparison: Codex + GPT-5.5 on GameCraft-Bench over 3 passes scored 71.52 with HoH vs 58.24 by simply continuing the same agent.
- Stress test: 70 loops produced a playable FPS from high-level requirements — though only one game project with extra tools, so generalization remains open.
- Takeaway: for long-running agents, invest in persistent project state, independent QA, and evidence-driven replanning — not just more calls.
More from coding & agent
- Grok launches Imagine Video 1.5 agent powered by Image 2.0 for multi-shot storytelling — belce_dogru · 2026-09-06
- Pamela Fox: I like LLM-generated code, but give READMEs a human pass — DanWahlin · 2026-09-06
- Taking a Break Is Hard When Your Astra Agent Keeps Running /goal — AIandDesign · 2026-09-06
- Dev Powers Filesystem Simulator VSH With Monty to Track Side Effects Before Execution — samuelcolvin · 2026-09-06
- Cheating Agents Answer Faster: A Missed Red Flag for Reward Hacking — zainhas · 2026-09-06
- First WebMCP benchmark: 3-5x faster, up to 23x cheaper, +11.6% task success vs computer use — laparisa · 2026-09-06