AI agents with $10,000 budgets still fail open-ended research: GPT-6 Astra and Fable 5.1 papers rejected
sayashk · x · 2026-10-21
An update from Sayash Kapoor and colleagues on evaluating AI agents doing open-ended AI research, beyond narrow verifiable tasks.
- Setup: agents get research questions from unpublished papers plus thousands of dollars in API credits and compute; original paper authors review the AI-written papers NeurIPS-style
- July: papers by Claude Opus 4.8 and GPT-5.6 Sol agents were both rejected
- September follow-up with GPT-6 Astra and Claude Fable 5.1 at a $10,000 budget: clear gains in experimental design and resource management, but persistent weaknesses in creative problem solving, judgment, calibration and communication; both papers were unambiguously rejected (overall scores of 1-2 out of 6 across all four papers)
- The team plans to continue with better models, scaffolds and higher budgets
Takeaway: even the newest frontier models remain far from producing publishable original research autonomously.
Related event: Papers by frontier AI agents rejected by expert reviewers(2 posts)→
More from Research
- Three JEPA variants, one question: how well must a machine predict to plan? — TheTuringPost · 2026-10-21
- COLM 2026 paper shows low-quality training data gives LLMs 'brain rot' — _vztu · 2026-10-21
- Stanford Virtual Embryo Challenge partners with Eigen Labs' open research platform Yukon — anshulkundaje · 2026-10-21
- Laude Institute backs WaggleHaus, an open-source query engine for agentic queries over data lakes — lateinteraction · 2026-10-21
- Training linear-attention Laplace 3B on AWS Trainium2: 120K tokens/sec, fixed 29.5MB state — AWS ML Blog · 2026-10-21
- UC Berkeley open-sources SkyDiscover, an AI framework that discovers algorithms — CShorten30 · 2026-10-21