Study finds no universally best harness for automated discovery systems
LChoshen · x · 2026-07-22
A controlled study finds there is no universally best harness for automated discovery.
- Simpler harnesses often matched or beat more complex systems such as OpenEvolve.
- The best choice changed across model–problem pairs, so harnesses do not generalize reliably.
- The authors ran repeated budget-matched trials, strong baselines, and statistical hypothesis testing to handle the high-variance setting.
- Letting the system choose among harnesses online as rollouts progress can reduce the generalization problem and motivates adaptive discovery systems.
Related event: Study Finds No Universal Optimal Harness for Automated Discovery(2 posts)→
More from Research
- AI folks, stop saying "biology" when you mean drug development — KevinKaichuang · 2026-10-26
- Flow Reasoning Models turn flow matching into iterative reasoners, hitting 99.5% on Sudoku-Extreme — alec_helbling · 2026-10-26
- Surprising find: Alzheimer's-linked amyloid fibrils deliver RNA to sperm for embryo translation — EricTopol · 2026-10-26
- HarnessMM cuts agent tokens per task by up to 88% while raising success rates — omarsar0 · 2026-10-26
- Ben Recht: The world needs a truly open language model, and it's easier to build than you think — beenwrekt · 2026-10-26
- New Blog Breaks Black-Box Distillation Into Five Practical Steps — chrisrohlf · 2026-10-26