Harness code becomes a research battleground: CMU rewrites it with RL, Apple says it doesn't matter
burkov · x · 2026-10-13
ML researcher Burkov rounds up three diverging research takes on evaluation harness code for agents:
- CMU & Johns Hopkins: use an LLM + RL to automatically rewrite your harness's code to make it more effective for your task types;
- Google, Stanford, et al.: regularize the harness-rewriting process to avoid overfitting to training examples;
- Apple: with a strong enough LLM, the harness's code doesn't matter at all.
Same question — is the harness a variable or noise — with three different answers.
More from coding & agent
- Ramp CTO to Discuss Internal Coding Agent Behind ~75% of Merged PRs — graceisford · 2026-10-13
- Meta Muse gains access to approved company tools via Merge's governed MCP endpoint — shensi · 2026-10-13
- Build your own Jev-style decision model in 15 minutes with open models — Arindam_1729 · 2026-10-13
- NVIDIA open-sources SMLIB, its NURBS-based CAD kernel, after 20-year partnership — Promptmethus · 2026-10-13
- LLMs are System 2: Austin LangChain's framing for pairing fast judgment with slow reasoning — colinmcnamara · 2026-10-13
- A feed of next actions is the only sane way to manage massively parallel agents — _AustinCalvert_ · 2026-10-13