MLS-Bench: AI Agents Can Optimize ML Experiments But Fail to Discover New Methods
机器之心 · wechat · 2026-08-12
Researchers from UC Berkeley, Tsinghua, and other institutions introduce MLS-Bench, a benchmark designed to evaluate whether AI agents can discover genuinely novel machine learning methods. Existing evaluations often fail to distinguish whether a score improvement comes from algorithmic innovation or mere engineering tweaks like hyperparameter tuning.
Evaluation Design and Core Mechanism
- Task Scale: Includes 140 real research tasks across 12 domains (e.g., pre-training, RL, robotics), requiring agents to modify specific research components like optimizers or training objectives.
- Controlled Environment: By locking data pipelines, model capacity, and evaluation protocols, the benchmark ensures that performance gains are strictly attributable to the algorithm itself rather than engineering tricks or parameter scaling.
- Generalization Verification: Each task includes multiple test conditions to verify that a candidate method generalizes across datasets, model sizes, and environments, preventing single-instance overfitting.
Key Findings: Strong Optimization, Weak Discovery
- The initial five frontier models tested all fell short of Human SOTA. Expert analysis reveals that agents excel at local modifications and recombining existing components but rarely propose novel mechanisms that explain current failures and offer stable gains.
- Limits of Test-Time Scaling: While increasing iterations and sampling improves simple tasks, returns quickly saturate and often lead to overfitting on visible feedback. Agents still struggle with planning effective experiments and allocating limited compute budgets.
The benchmark has already been adopted in official evaluations for models like Kimi K3 and Qwen 3.8-Max.
Related event: Researchers Introduce MLS-Bench to Evaluate AI Research Capabilities(2 posts)→
More from Models
- Anthropic's Watermark Strategy Flawed: Could Become Top Distillation Target — cocktailpeanut · 2026-08-12
- Research Reveals the Personality Evolution of the Grok Model Family — DevDminGod · 2026-08-12
- User Slams OpenAI's Safety Filters While Auditing Insulin Pump — max_paperclips · 2026-08-12
- DeepSeek V4 Flash Jailbroken Using Copied Gemma 4 Prompt — GodComplecs · 2026-08-12
- Users Accuse Anthropic of Cooked Evals, Claiming Real API Performance Lags — GabGarrett · 2026-08-12
- Encrypted Chain-of-Thought in Proprietary LLMs Can Be Extracted via Weaker Sibling Models — Simon Willison · 2026-08-12