Can AI Optimize AI? New Benchmark Tests Recursive Self-Improvement
shehio · reddit · 2026-08-28
Scale released a study testing whether LLMs can improve other AI agents by rewriting their code harnesses. To prevent cheating (like the OpenAI eval agent incident), they introduced HarnessOpt-Bench, which isolates the optimizer in a sandbox with no access to test data or API keys.
Key Findings:
- Model Evolution: On one task, GPT's optimization capability rose from 3% to 49% of headroom between Nov 2025 and Jul 2026; Claude Opus rose from 37% to 59%.
- No Home-Field Advantage: Models don't necessarily perform best in their native tools (e.g., Claude Code). The third-party tool OpenCode beat native harnesses in 11 of 20 model–task pairs.
- Model > Harness: The choice of model impacts performance gains 1.8× more than the choice of harness.
Related event: Scale's HarnessOpt-Bench Measures Recursive AI Self-Improvement(2 posts)→
More from Research
- GUI-Primitives Benchmark Reveals VLM Spatial Reasoning Failures — usc-isi · 2026-08-28
- Factory releases ProgramBench: A benchmark for reproducing real software from scratch — matanSF · 2026-08-28
- Ablation discussion: query head count barely matters early, sparse routing must be learned — stochasticchasm · 2026-08-28
- 404 Team to Walkthrough Titan Model and World Models — markjeffrey · 2026-08-28
- NVIDIA Details QAD Pipeline for Optimizing Nemotron Model — PyTorch · 2026-08-28
- Study: AI Coding Agents Are Reshaping PR Vocabulary — ycombinator · 2026-08-28