CentaurBench Reveals Strongest Models Aren't Always Best Assistants
soumitrashukla9 · x · 2026-08-27
UC Berkeley released CentaurBench, a new benchmark evaluating LLMs' "augmentation" capabilities—assisting a fixed worker model—versus traditional "automation" mode.
Key Findings:
- Model rankings differ significantly between automation (direct execution) and augmentation (providing assistance).
- For instance, Opus and Sonnet excel at automation but lag in assistance; Gemini models are stronger assistants than automators; GPT-5-Mini performs well in both.
Methodology:
- Evaluated across 7 real-world economic tasks derived from GDPval and Anthropic's Economic Index.
- Outputs are scored blindly using a validated LLM-as-judge framework.
- Top augmenters share three traits: defining what constitutes "good" work, structuring reasoning steps, and reinforcing task requirements.
Related event: CentaurBench: The Strongest Models Aren't Always the Best Assistants(3 posts)→
More from Research
- OpenAI gives $5M to Simons Foundation's $15M push to help mathematicians navigate AI-era results — OpenAI · 2026-10-15
- DeepPhylo infers phylogenetic trees without unrealistic evolutionary assumptions — anshulkundaje · 2026-10-15
- Wander preprint explores graph foundation models via shared objective and random-walk interface — mmbronstein · 2026-10-15
- CHI Reviewer: AI Slop Is Real, but Unusually Rigorous Papers Still Getting Through — IanArawjo · 2026-10-15
- Into the Unknown: CMU benchmark finds 114 of 135 AI-written research papers fail integrity checks — arankomatsuzaki · 2026-10-15
- Benchling launches Science Foundries: multiplayer agent swarms for end-to-end science workflows — nlarusstone · 2026-10-15