CentaurBench: New Benchmark Evaluates Whether LLMs Augment or Automate Human Work
random_walker · x · 2026-10-16
Arvind Narayanan and Gabriel Unger argue the AI industry's "automate everything" agenda stems not just from economic incentives but from deep cultural issues in computer science — and that this can change.
Key points:
- The essay sketches four research directions needed for collaborative/augmentative AI that the automation paradigm largely ignores
- The companion CentaurBench benchmark evaluates which models most improve the work of another (weaker) agent, rather than just ranking automation ability
- Tested across seven economically grounded real-world tasks, with an assistant model writing assistance text for a standardized weaker worker model; outputs scored via blind pairwise comparisons by an LLM judge panel
- Nagaraj (UC Berkeley) also shares a multi-dimensional framework for measuring AI impact: task type, augmenting vs replacing, point vs system, which AI, and user skills
Related event: Princeton's Narayanan Calls for AI to Augment Humans Over Automation(4 posts)→
More from AGI Musings
- Robotics' hidden data problem: the undervalued opportunity of a deployment context layer — broodsugar · 2026-10-16
- RFdiffusion3 published in Science, alongside Chaperone agent harness for bio-molecular design — anshulkundaje · 2026-10-16
- Prediction: No New Jobs for Humans by 2028, Everyone Unemployed by 2030 — davidpattersonx · 2026-10-16
- Debate erupts over Apollo Research: is self-alignment a precursor to malign evolution? — teortaxesTex · 2026-10-16
- Mathematicians Weep: Fields Medalist Mourns as OpenAI Cracks Decades-Old Proofs — stevenstrogatz · 2026-10-16
- Why does matrix multiplication on silicon 'not count' as thinking, but sodium ions do? — khademinori · 2026-10-16