ScientistTwo: Pioneering the Human Knowledge Frontier with Autonomous AI
Jaehyun Nam, Jinsung Yoon, Yanzhou Pan, Yubo Wang, Rui Meng, Parthasarathy Ranganathan, Tomas Pfister
cs.AI
2026-09-17
ScientistTwo takes a top-venue paper as the problem, then autonomously experiments, writes, and simulated-reviews. It succeeds on 86 of 107 tasks, averaging +25.2% over human SOTA.
Autonomous research agents still stall at two places. Most of them optimize a single scalar on a single dataset, which produces a better score rather than a paper that could survive a top-venue review. They also skip the closed loop working scientists actually run: after the main experiment they stop, they do not ablate components, and they do not answer reviewer criticism with new runs.
ScientistOne already emits a paper plus code, yet ScholarPeer scores it 3.8 on average and the Stanford Agentic Reviewer accepts none of its outputs. ScientistTwo, from Google Cloud AI Research, raises the target. A human supplies a problem, typically an already-accepted ICLR, ICML, or NeurIPS paper with its codebase. The system is then supposed to find that paper's weaknesses, invent a method, run the full empirical stack, write a manuscript, and leave a reproducible repository.
The pipeline is a set of specialist agents with a critic at every stage. Accept, refine, or discard.
A limitation extractor loops over the source paper until a verifier says the weakness list is rich enough to drive new ideas. An idea generator then proposes seeds around those weaknesses, scores novelty against two papers retrieved via Google Search, and ranks the pool.
A coding agent first reproduces the human baseline on a data subset, then implements each candidate. The critic returns Bad, Good, or Engineer. Weak ideas are dropped. Borderline ones get at most two engineering rounds. Survivors scale to the full benchmark. Evolution then mixes two sources: an Idea Evolver that reads both successful and failed traces, and leftover high-novelty seeds that have not been tried yet. At most four rounds; the loop stops early once four successful ideas exist.
After a selector picks a winner, an ablation planner tears the method into components. If a piece does nothing, the system rewrites once and keeps the rewrite only if full-set numbers actually improve. Drafting uses PaperOrchestra in ICLR 2025 format. ScholarPeer scores the draft. Below 8, a rebuttal planner invents supplementary experiments, a coding agent runs them, and a writer patches tables and claims. At most two review rounds. A meta-reviewer can still send the idea back one more time, which restarts ablation and drafting.
The default brain is Gemini 3.6 Flash. Code, ablation, rebuttal, and draft enhancement go through Claude Code with Opus 4.8.
The test bed is 107 tasks: 38 NeurIPS 2025 acceptances, 5 ICLR 2026 acceptances, and 64 ICML 2026 Spotlights. 86 succeed, an 80.4% hit rate. Relative gain over the human SOTA averages 25.2%, but the median is 7.7%. A handful of blowouts pull the mean; the median is the more honest typical step.
Two AI reviewers sit on the papers. ScholarPeer is used inside the writing loop, so it is in-distribution. Stanford Agentic Reviewer was held out during development.
| System | ScholarPeer mean / accept | Stanford mean / accept |
| ScientistOne | 3.8 / 14.3% | 4.1 / 0% |
| ScientistTwo | 7.5 / 91.9% | 5.7 / 72.1% |
| Human ICLR 2026 accepted | 6.8 / 60% | 5.2 / 60% |
| Human NeurIPS 2025 accepted | 6.2 / 65.8% | 5.5 / 76.3% |
| Human ICML 2026 Spotlight | 6.9 / 79.7% | 6.1 / 96.9% |
Against AutoSOTA, which edits existing code to chase one metric, ScientistTwo finishes fewer tasks (86 vs 105) but posts larger average gains on successes (25.2% vs 7.5%). On the four overlapping ICLR 2026 tasks the ranking flips: AutoSOTA 7.2%, ScientistTwo 3.8%. Appendix B reports that AutoSOTA's five ICLR edits are configuration tweaks of a few lines, with no new algorithmic module.
Without simulated review, first drafts already reach about 47-49% acceptance on both reviewers. One rebuttal round lifts Stanford acceptance to 73.5%. A second round pushes ScholarPeer to 93.9% while Stanford slips to 69.4%. Overfitting to the in-loop reviewer has started.
An integrity audit on 49 papers reports 49/49 reproducible scores, 0 specification violations, 0 hallucinated citations, and 49/49 method-code alignment. Drop the matching repair agents and hallucinated references jump to 19, while alignment falls to about 38/49.
Nine human reviewers scored 33 NeurIPS-derived manuscripts on a 1-5 Likert scale. Standalone scores sit above 3.0 on every axis, 3.7 overall. Head-to-head with the human accepted papers: overall parity (3.0). Experiments lean toward ScientistTwo (3.3-3.5). Method soundness still leans human (2.9).
On those 33 NeurIPS tasks the mean wall time is 2.5 days and the mean bill is $3,765 in tokens plus VMs, rounded to about $3,800 in the conclusion. Most of both is idea evolution, experiment execution, and the review loop.
One compounding case is clean. On incremental BPE, VD-STrans beats the human SOTA by 10.9%. Fed back as the new prior, BXT-Transducer adds 9.6%. A third cycle, SBR-Transducer, adds 8.2%.
This is one of the most complete attempts to turn autonomous research from score chasing into a conference-style package: multi-dataset evaluation, ablations, reviewer-driven extra experiments, citation hygiene, and code alignment. For an ML problem that already has a reproducible codebase, two to three days can yield an incremental paper that AI reviewers rate near accepted work.
Do not read it as open-ended discovery. The task is to push an already-accepted paper one step further. Human reviewers still prefer the human method section. Spotlight-level Stanford scores are not reached (5.7 vs 6.1, 69.4% vs 96.9% accept). The 25.2% mean gain is the wrong headline; 7.7% median is the typical move.
At about $3,800 a run, this is not a lab default. An industrial group hunting a specific improvement can try it. Swapping the coder to Antigravity with Gemini 3.8 Flash drops success from 80% to 60% on five ICLR tasks, so Opus-class coding still carries a lot of the system.
The authors say two things themselves. The work does not yet hit Spotlight or Oral conceptual jumps. And roughly $3,800 per task prices out most academic groups.
The other caveats sit in the tables. ScholarPeer's 91.9% accept rate is the same reviewer family grading its own drafts. Stanford is still an AI reviewer, not a program committee. 21 of 107 tasks fail. Success means "beat the source paper," not "would be accepted if submitted." AutoSOTA's higher ICLR gains show that a new method is not always worth more than tuning the old one. Mean far above median means a few outliers. The problem set was filtered for reproducibility, so it already leans engineering-friendly.