A controlled study finds no universal best harness for automated discovery
LChoshen · x · 2026-07-23
Automated discovery does not have a universally best harness
A study shared in the thread reports that no single harness was reliably best for automated discovery across model–problem pairs.
Key findings:
- Harness choice behaves like a model/problem-specific hyperparameter, not a universal recipe.
- Simpler harnesses often matched or beat more complex systems, including OpenEvolve variants.
- The best harness changed depending on the model–problem pair.
- Because this setting is high variance, the authors used repeated budget-matched runs, strong baselines, and statistical hypothesis testing to compare methods robustly.
The practical takeaway is that online harness selection may matter more than adding complexity to the discovery system itself.
Related event: Study Finds No Universal Optimal Harness for Automated Discovery(2 posts)→
More from Research
- New review: how prefrontal cortex dynamics turn world models into flexible behavior — AnnaCiaunica · 2026-10-27
- LIBERO-MAX: Robot Policies Drop 11–26 Points When the World Changes Mid-Task — drmapavone · 2026-10-27
- Kolab Benchmark Uses Pro Screen Recordings to Show AI Agents Overestimate Collaboration Quality — minnesotanlp · 2026-10-27
- One BBQ Example Pushes LLM Fairness Scores Past 90%, EMNLP Paper Questions Benchmarks — radamihalcea · 2026-10-27
- Real2Sem2Real turns one phone video into 2,000+ robot demos, matching 150 teleop demos — Scobleizer · 2026-10-27
- Understanding MuonH (Hyperball) through weight norm geometry of scale-invariant networks — jasondeanlee · 2026-10-27