FrontierML benchmark: Fable 5.2 leads autonomous ML research at 0.523, no model near target
rohanpaul_ai · x · 2026-11-19
Synthetic Sciences released FrontierML: 25 open-ended ML research problems judged by stated criteria with no narrow metric to optimize, rated 9.5/16 by METR for realism. Each run is 12 hours in a virtual lab where the agent iterates on frozen literature/data, model APIs, and per-job rented H100s paid from a $50–125 wallet; sealed submissions prevent data leakage.
- Fable 5.2 leads with a mean normalized score of 0.523; Opus 5.5 scores 0.399, Astra 0.371, Carbon 0.353, Grok 4.7 only 0.115 (public tasks only). No run reaches the strong-result reference.
- Lab0 adds composite taste metrics (metacognition, novelty, efficiency, control); no model hits two-thirds of target on any.
- Models split between judgment and invention: Opus 5.5 judges its results best, Fable 5.2 invents most mechanisms and spends budgets best. Ablations show simplifying tasks doesn't help; only new mechanisms pay after a strong result.
Related event: FrontierML Benchmark Launches, No Frontier Model Passes(4 posts)→
More from coding & agent
- Vercel: agents need "headless Slack" — sync major state, don't spam humans — cramforce · 2026-11-20
- Ex-OpenAI CTO: fine-tuned SLMs can beat frontier models on specific micro-tasks — ThePeterMick · 2026-11-20
- Building a WebGPU pinball machine with Gemini: 480Hz physics, zero audio samples — AI_Andrew · 2026-11-20
- Claude autonomously rents a 4x RTX A6000 pod and resumes training unattended — vxnuaj · 2026-11-20
- Reversing a $40M funnel with 6 Claude Code agents in 8 minutes — aryanXmahajan · 2026-11-20
- Asked Claude to add screw holes in CAD, got this look instead — _Stocko_ · 2026-11-20