ScientistTwo AI runs 107 research tasks itself, improving 86 papers at 80% success
AGI Hunt · wechat · 2026-09-22
Google Cloud AI Research's ScientistTwo takes a paper plus code, then autonomously proposes improvements, runs experiments, writes papers, and ships executable code. On 107 tasks drawn from NeurIPS/ICLR/ICML papers (AutoSOTA selection criteria), 86 (80.4%) reported improved task success, with 25.2% average relative gain (7.7% median).
The pipeline: novelty checks via web search, subset experiments before full runs, AblationPlanner to trace where gains come from, and a PaperOrchestra of two AI reviewers handling drafting and final review. Gemini 3.6 Flash does planning/ideation/reviewing; Claude Code + Opus 4.8 implements experiments. Swapping in Antigravity + Gemini 3.8 Flash cut success to 60% but raised average per-task gains.
Costs and quality: 2.5 days and $3,765 per task on average. Nine human reviewers scored 33 generated papers at parity (3.0/5) with the human originals on overall scientific maturity—broader experiments, slightly weaker methodological rigor. An integrity audit found all 49 audited outputs reproducible, zero hallucinated citations, zero rule violations. Caveats: high cost, modest typical gains, and self-improvement over three rounds was only validated on one BPE tokenization case.
Related event: Google's ScientistTwo Automates Research at 80% Success Rate(2 posts)→
More from AGI Musings
- The worst AI use signals you don't care about things you should care about — Afinetheorem · 2026-11-12
- Worst AI use: signaling you don't care about things you should — Afinetheorem · 2026-11-12
- Stanford HAI blueprint: 56 US agencies report 3,611 AI use cases, evaluation lags — StanfordHAI · 2026-11-12
- Fleuret amplifies debate: AI can't slash wet-lab experiments in unmapped biology — francoisfleuret · 2026-11-12
- Recursive bets on open-endedness: agents running massive parallel experiments like scientist communities — agihouse_org · 2026-11-12
- Recursive's Tim Shi maps recursive self-improvement: from learning data to discovering architectures — agihouse_org · 2026-11-12