Automating ML for Science: Can Frontier Agents Climb Scientific Hills in the Wild?
Ming Zhong, Stacy Li, Nicholas Carlini, Matthew Jagielski
NeurIPS 2026 E&D Track
cs.AI, cs.CL, cs.LG
2026-10-21
Four agents with 8 H200s and 12 hours beat the published best on all 20 science benchmarks. Opus averages a 10.4% gain, 20.1% with plug-in skills, mostly via ML engineering.
Coding agents are often described as closed-loop researchers. Most public demos still sit on tidy repositories, one task, and computer-science targets. ML for science is messier: the repo is usually a paper artifact, and lifting the metric tends to require knowing what the domain is actually constraining.
Handing an agent the raw repo also fails a basic honesty check. On 12 tasks from 5 NeurIPS 2025 science benchmarks, 5 showed serious evaluation gaming: reporting a favorable subset, peeking at the test set, rewriting the metric, or hardcoding outputs. Scientific repos often lack a fixed split and a full training script, so an aggressive optimizer finds the seam even when the prompt is earnest.
Agent4Sci asks how far today's frontier coding agents can push a published scientific model with no human in the loop, and how much of that push is domain insight.
The task is scientific hill-climbing: start from the best model already in hand and push one metric under a fixed budget. The agent gets the paper, code, data, and target metric, then edits code and runs experiments for 12 hours on 8 H200 GPUs. A person only starts the run, stops it at the limit, and collects artifacts. Four agents run through their official CLIs at maximum reasoning effort: Claude Opus 4.6, Claude Sonnet 4.6, Codex (GPT-5.4), and Gemini 3.1 Pro.
Before any of that, each task is standardized by hand. It enters the suite only if the paper's headline number can be reproduced. The test set is sequestered, so hill-climbing sees train and validation only, and the baseline is squeezed onto one H200. The NeurIPS 2025 Datasets and Benchmarks track shows the filter: 497 papers, 10 kept. The suite is 20 benchmarks, 25 tasks, 11 domains. Three benchmarks needed a training pipeline the release did not ship; two needed a validation slice cut from training data, because checkpoint selection cannot touch the test set.
Each run is split in three. The hill-climbing agent submits code and checkpoints. A second agent scores them on the held-out test set in an isolated environment. A third audits the trajectory for selective reporting, test-set leakage, metric changes, and hardcoded outputs. Relative gain is (new − baseline) / |baseline|, with the numerator flipped when lower is better, so a positive number always means improvement. Baselines are reproductions. CarbonGlobe's RMSE moves from 0.571 in the paper to 0.604 in the reproduction because the original protocol picked checkpoints on the test set; the reproduction uses a 9:1 split inside the training set.
Three failures showed up often enough to prompt three short markdown skills, loaded at a set point in the run. Research ideation makes the agent read domain literature and treats hyperparameter tuning as beneath it. Experiment management requires live job monitoring, GPU time for the bolder experiments, and less CPU/GPU contention. An hourly checklist asks whether the current work is a new experiment or another knob turn. The failures they target: cosine learning-rate schedules presented as creative ideas, idle GPUs, and a quick collapse into seed and learning-rate sweeps once some change works.
Every benchmark was beaten by at least one agent, and the audit flagged no academic misconduct. Half of each agent's runs were re-audited by three other models; scores and verdicts agreed. Auditors from all three model families also caught every known cheat in the pilot trajectories.
Mean relative gain across the 20 benchmarks is 10.4% for Opus, 9.6% for Sonnet, 6.8% for Gemini 3.1 Pro, and 4.4% for Codex. The spread is wide. On ProteinGym the best default agent, Sonnet, gains 1.1% over Kermut, already the public leaderboard leader. On PFDelta, Opus cuts average power-balance loss (PBL) by 64.4%, from 0.39 to 0.14, by adding a gradient-safe squared PBL as an auxiliary loss, raising the MSE weight λ from 0.1 to 0.3, and switching to cosine annealing with warmup. On Mars-Bench cone detection, Opus swaps RetinaNet for Faster R-CNN and lifts mAP from 16.5 to 30.4.
The moves mostly match. Opus puts 56% of its changes on hyperparameters and 44% on architecture. Codex is more spread out: hyperparameters 44%, and feature engineering, data augmentation, and loss design at 24% each. Gemini sits at 48% hyperparameters and 32% architecture. Problem-specific ideas stay rare.
Workflows split three ways. Opus stays on the run for the full budget, 12.0 hours on average, with 490 turns and 338 tool calls. Codex writes one long script in the first hour and waits: 328 turns and 325 tool calls packed into 1.0 hour of active time, enough to launch over a hundred experiments without reading intermediate results. Gemini switches between the two, at 196 turns and 201 tool calls.
The three skills move Opus from 10.4% to 20.1% (18 of 20 benchmarks improve) and Codex from 4.4% to 8.1% (12 of 20). On PFDelta, Opus goes from 64.4% to 91.0%, and PBL from 0.39 to 0.04. The agent treats PBL as a label-free objective implied by Kirchhoff's laws, trains on it as the primary loss, and at inference uses the graph-network prediction as a warm start for the AC power-flow equations, driving the voltage residual toward zero. GEO-Bench-2 mIoU gain rises from 9.6% to 37.9%, TIDMAD from 7.4% to 30.0%. Skills also hurt: Codex with skills goes to −12.8% on PFDelta and from +4.8% to −10.0% on LITHOS.
| Setup | Mean relative gain, 20 benchmarks | Compared with |
| Claude Opus 4.6 | 10.4% | reproduced published best |
| Claude Sonnet 4.6 | 9.6% | same |
| Gemini 3.1 Pro | 6.8% | same |
| Codex (GPT-5.4) | 4.4% | same |
| Opus + skills | 20.1% | 10.4% without skills |
| Codex + skills | 8.1% | 4.4% without skills |
Agent4Sci is a probe. A second long run can land elsewhere because the path through edits is stochastic, so the mean is a weak ranking. The useful signal is whether a new agent leaves a move that transfers to another scientific task.
For someone with a reproducible science repo, 8 GPUs and 12 hours mostly buy backbone swaps, auxiliary losses, and learning-rate schedules. Opus averages 10.4% that way. When a physical residual can enter the loss, as on PFDelta, a markdown skill can push the same agent from tuning λ to solving against that residual. The evaluation protocol stays put.
Against MLE-bench and RE-Bench, these tasks come from published code in 11 scientific domains, and every trajectory is audited. Automated research here still looks like a very persistent ML engineer.
Only default single-agent harnesses were tested. Multi-agent search setups are out of scope, so 10.4% is an out-of-the-box number. Held-out scoring blocks the four gaming behaviors from the pilot and does not block subtler rationalization. Hill-climbing from a finished baseline does not measure choosing a problem. There is no human-expert comparison. On ProteinGym the gain over public-leaderboard leader Kermut is 1.1%.
The skills were written after reading these failures. They name no benchmark, but they can still overfit the 11 domains and Claude Code's habits. The similar proportional gain on Codex shows only partial transfer. On a harder PFDelta variant, Codex with skills ran for 3 days and reached a near-perfect solution; the main text does not report that score.
The mean is pulled by large movers such as PFDelta and Mars-Bench. The best default gains on gRNAde and SR-CACO-2 are 2.6% and 1.4%. The appendix measures run-to-run variance on six benchmarks and the main text gives no standard deviation, so 10.4% versus 4.4% should not be read as a stable ranking. Reproduction, two-reviewer checks, and isolated containers were finished by people before any agent started.