New Benchmark Shows Best Automating Models Are Not Best Coaches
soumitrashukla9 · x · 2026-08-27
The paper CentaurBench introduces a framework evaluating LLMs on two distinct modes: Automation (doing the work directly) vs. Augmentation (coaching a weaker agent/human).
Key Findings:
- Rankings Diverge: The top-performing model in automation loses in augmentation mode on 5 out of 7 real-world tasks.
- Assistance Can Hurt: Model guidance is not reliably positive. In 3 tasks, the unaided worker outperformed every assisted condition. Only one model provided guidance that beat having no guidance on average.
- Implication: Automation capability is an incomplete proxy for assistance quality, necessitating new benchmarks for human-AI and multi-agent collaboration.
Related event: CentaurBench: The Strongest Models Aren't Always the Best Assistants(3 posts)→
More from Research
- Schmidt Sciences awards $10M to 19 AI humanities research teams — OdedRechavi · 2026-10-15
- Mistral and CERN partner to build a Large Physics Model trained on billions of simulated collisions — MistralAI · 2026-10-15
- Global Transport coupling lifts guided flow generation: ImageNet FID 2.06 to 1.91 — mmbronstein · 2026-10-15
- Your breathing gives you away: Wi-Fi signals can reveal your breath rhythm — TinfoilTricorn · 2026-10-15
- Beyond text correlation: attaching a geometry simulator to ground model reasoning — gerardsans · 2026-10-15
- Sequen CEO Zoe Weil: Ranking Models Should Optimize Purchases, Not Clicks — TWIML AI Podcast · 2026-10-15