Meta AI's AutoBenchmark auto-creates benchmarks, finds human feedback gives big wins over agents alone
jaseweston · x · 2026-09-30
Meta AI researcher Jason Weston introduced AutoBenchmark: an agentic system that creates benchmarks automatically — specifically benchmarks that evaluate autoresearch agents themselves — closing a full recursive-improvement loop, while studying the role of humans in the loop.
Key findings: human-agent collaboration substantially beats agents alone, with fine-grained human feedback during the ideation stage crucial; autoresearch benchmarks for AI research can be built with this recipe; and benchmark creation works best with two feedback sources — benchmark solvers plus external verifiers (human + AI).
Related event: Meta AI introduces AutoBenchmark for automated benchmark creation(3 posts)→
More from Research
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16
- Berkeley paper: LLMs answer with stale values even when the update is in context — ombre-ai · 2026-11-16
- Open-source TotalSegmentator spots a 42mm dilated aorta on a routine chest CT — MaziyarPanahi · 2026-11-16
- Tivadar Danka Publishes 4,000-Word Visual Guide to Graph Theory for Machine Learning — TivadarDanka · 2026-11-16
- METR time-horizon chart updated: doubling every 1.7 months since Opus 4.5, 'Model 4' ~38h — ben_j_todd · 2026-11-16
- GTAgentClinic: one-line install, plug in Claude Code or Codex for a 3D hospital shift — Michael_D_Moor · 2026-11-16