FrontierML benchmark grades AI agents on autonomous ML research; Claude Fable 5.2 leads at 0.523
rohanpaul_ai · x · 2026-11-19
FrontierML released its technical report and benchmark evaluating frontier models on autonomous empirical ML R&D across 25 graded tasks (10 private), scored 0 (starting reference) to 1 (strong-result reference, unreached).
- Leaderboard: Claude Fable 5.2 tops at 0.523 mean, Claude Opus 5.5 0.399, GPT-6 Astra 0.371, Gemini 4 Carbon 0.353; Grok 4.7 ran only public tasks at 0.115.
- Beyond task scores, four composite "research taste" measures are built from every job, model call and hand-in: metacognition, novelty, efficiency (progress per research dollar), and control (budget converted to working experiments).
- No model reaches two-thirds of any target and none leads on all four; metacognition peaks at 0.35, control at 0.63.
- The report includes task examples (Pokémon Battles, Rule Worlds, Robot Reward, etc.) and a public leaderboard for digging into agent traces.
Related event: FrontierML Benchmark Launches, No Frontier Model Passes(4 posts)→
More from Research
- Researchers show VLMs plan via image pointers before generating a single token — gsarti_ · 2026-11-20
- NeurIPS 2026 to host full-day UserSim workshop on trustworthy user simulation — alexisjross · 2026-11-20
- e-means' two counterexamples fixed in 10 minutes with Astra 6.2; high schooler solves 6 conjectures — amuellerml · 2026-11-20
- seq2fun assumes fixed cell state; transcriptome-only virtual cell models ignore DNA regulation — anshulkundaje · 2026-11-20
- The Economist: Will AI Make Economics Literature More or Less Reliable? — TuhinChakr · 2026-11-20
- Paper Revisits How Longshoremen Unions Faced the Containerization Shock — Afinetheorem · 2026-11-20