GPT-6 Astra and Claude Fable 5.1 write $10,000 research papers — experts reject all
sayashk · x · 2026-10-21
- Evaluator pkirgis gave GPT-6 Astra and Claude Fable 5.1 each a $10,000 budget to tackle open-ended research questions from unpublished papers, then had the original authors grade the resulting papers.
- Versus July tests with Claude Opus 4.8, agents showed meaningful gains in experimental design and resource management, but persistent weaknesses in creative problem solving, judgment, calibration, and communication.
- Per the shared expert-review chart, all four papers (two in July, two in September) were rejected with overall scores of just 1-2 out of 6; the September rerun of the same question scored higher on soundness but lower on presentation and contribution.
More from Models
- ChatGPT desktop app crashing every few minutes, killing users' agents — wstone_bd · 2026-10-21
- Polish dev defends open model: built on Gemma with more than a simple LoRA fine-tune — ColinWPLewis · 2026-10-21
- X trends: SophontAI's Holon-S1 predicts gene activity from cancer slides; decision models rise — iScienceLuvr · 2026-10-21
- OpenAI's MultiLangBench: PHP scores highest (91.3%) at building a database engine from scratch — ricklamers · 2026-10-21
- 704-vote robotics Elo ranks Claude Opus 5.5 first at 1379; no model completes the task — qinzytech · 2026-10-21
- DeepSeek V4 called 'god-tier infra, 4th-tier ML': 7 tricks stacked, 20 patches to fix them — teortaxesTex · 2026-10-21