CRUX shadow eval: GPT-6 Astra and Claude Fable 5.1 research papers both rejected by experts
patpat_mit · x · 2026-10-28
- CRUX gave GPT-6 Astra and Claude Fable 5.1 a $10,000 budget each to produce papers answering open-ended AI research questions from unpublished papers, then had the original authors review them. Both papers were unambiguously rejected.
- Compared to July's first tests (Claude Opus 4.8 / GPT-5.6 Sol), the new models showed meaningful gains in experimental design and resource management, but persistent weaknesses in creative problem solving, judgment, calibration, and communication.
- This is round two of CRUX's recurring 'shadow evaluation' project measuring frontier agents on messy, long-horizon open research, with collaborators including Arvind Narayanan and Nicholas Carlini.
More from Models
- Kimi K3.1 scores 38% on RareBench, matching GPT-6 Astra at a quarter of the cost — danielmckinn0n · 2026-10-28
- GPT-6 Astra hits SOTA on ARC-AGI-3, inventing its own programming language — typewriters · 2026-10-28
- Abliterated LLMs surge as open-source community strips refusal mechanisms — mishig25 · 2026-10-28
- User Burns Through Four Claude Max x20 Plans, Calls for a $500/Month Tier — andrey_kurenkov · 2026-10-28
- Claude autonomously finds zero-click Linux kernel RCE and dozens of working exploits — burny_tech · 2026-10-28
- Bindu Reddy: Argon Is Bad for Coding Due to Poor Steerability and Hallucinations — bindureddy · 2026-10-28