CentaurBench: Benchmarking LLM Capabilities on Augmenting vs. Automating Real-World Work Tasks
Pattaraphon Kenny Wongchamcharoen, Kris Gulati, Min Min Fong, Abhishek Nagaraj
cs.CY, cs.AI, cs.MA, econ.GN
2026-08-19
Ten models scored as solvers and as coaches to GPT-3.5-Turbo on 7 work tasks: rank correlation 0.48, winners flip on 5/7, and only GPT-5-Mini beats unaided 3.5 on average.
Most LLM leaderboards still score one thing: can the model finish the job by itself. Newer occupational suites such as GDPval and JobBench made the tasks look like real work, but the unit of evaluation is still an end-to-end deliverable. Field experiments in economics ask whether AI helps a worker write faster or handle tickets faster. They usually test one tool, one setting, against an unaided control.
A lot of actual use looks different. The model writes a plan, a checklist, a self-review. A person, or a cheaper executor model, owns the final document. Skill at the task is not the same as skill at coaching it. CentaurBench puts the same models on the same seven work tasks in both roles, and asks whether automation rank is a usable proxy for assistance rank.
The DIAL lab at UC Berkeley Haas picked seven tasks with observable deliverables: a constrained seven-day menu, a Tokyo itinerary under a budget, a natural-gas market brief, a logistics optimization memo, a tax discrepancy review, a counseling-style reply, and a Grade 3 lesson segment. Most prompts come from GDPval and the Anthropic Economic Index; the tax task is author-designed so rule errors can be checked.
Two modes run in parallel. In automation, each of ten models writes the deliverable. In augmentation, each candidate writes only an assistance text for a frozen worker, GPT-3.5-Turbo, which then writes the deliverable. The assistance prompt forces process scaffolding: a requirements check, an execution plan, a final checklist. The assistant is forbidden from producing the deliverable. What varies is guidance quality, not whether the strong model quietly did the work.
The candidate set is Claude-Opus-4.8, Claude-Sonnet-4.6, Gemini-3.1-Pro, DeepSeek-V3.1, GPT-5-Mini, GPT-OSS-120B, GPT-O4-Mini, GPT-4.1, GPT-O3-Mini, plus GPT-3.5-Turbo as the worker. Augmentation also includes a plain baseline: the worker with no assistance text. GPT-3.5-Turbo is the worker because it sat near typical performance on the pilot tasks, and because cheap, weaker models often occupy the executor slot in multi-agent stacks.
Scoring is a blind pairwise tournament. Four LLM judges (Claude-Opus-4.8, DeepSeek-V3.1, Gemini-3.1-Pro, GPT-4.1) use task-specific plus general rubrics, with leave-family-out so a judge never scores its own provider. The full pipeline is repeated ten times. Ranks come from win rates, then rank-of-ranks. Automation and augmentation never share a tournament.
The assistance protocol is a conservative lower bound: teach how, withhold the answer. The numbers describe this form of coaching, not draft-then-edit or error-correction.
On automation, GPT-5-Mini has the best average rank, then Claude-Opus-4.8. GPT-5-Mini leads menu planning, tax, travel, and tutoring. Claude-Opus-4.8 leads market trends. GPT-OSS-120B leads counseling. Gemini-3.1-Pro and GPT-OSS-120B jointly lead operations research. GPT-3.5-Turbo ranks last on average. This ordering is for tool-free, single-turn, bounded deliverables. It should not be read as an agent-with-tools ranking.
On augmentation, no assistant dominates. GPT-4.1 leads counseling, GPT-O4-Mini leads market trends, GPT-5-Mini leads menu planning and tutoring. On operations research, tax, and travel, unaided GPT-3.5-Turbo beats every assisted condition. Averaged over seven tasks, the plain baseline has mean rank 3.79. GPT-5-Mini is the only assisted condition with a better mean rank, 3.66.
Across the nine assistant models, Spearman correlation between the two roles is 0.48 (p=0.187), not distinguishable from zero. Task-level correlations range from 0.85 on tax (p=0.004) to -0.04 on travel. Five of seven tasks change winner across modes. Claude-Opus-4.8 on market trends: automation mean rank 2.05, augmentation 8.15. GPT-4.1 on counseling: automation 7.40, augmentation 3.80. Judges agree on the winner in 71.0% of 6,265 multi-judge comparisons (74.5% automation, 67.8% augmentation). The chosen winner has the higher mean rubric score in 99.7% of non-tied comparisons.
Reading the assistance texts, useful scaffolds tend to add an analytical step the prompt did not already contain, lock a section order, and keep the worker pointed at the deliverable. Weak scaffolds restate the prompt, leave structure to the worker, or explicitly ban the analysis the task requires. No assistance often beats bad assistance.
"Which model is best" is the wrong buying question. The useful question is which model for this role and this task. Finishing work and coaching work come apart; automation leaderboards are a weak proxy for the second.
Assistance can also go negative. Nine stronger models wrote process guidance for GPT-3.5-Turbo. On average, only one assisted condition beat "let 3.5 think alone." Putting a frontier model in the planner slot does not automatically raise downstream quality.
For anyone wiring multi-agent or human-in-the-loop stacks, the reusable piece is the measurement recipe: freeze the executor, swap only the coach, score both roles on the same tasks. Code and a dashboard are public. Do not treat the current ranks as a purchasing table. The worker is GPT-3.5-Turbo, coaching is one-shot process text, judges are LLMs. Change the worker, add turns, or let the coach supply answers, and the order can move.
The authors call this a pilot, not a capability map. Judges are LLMs; family hold-out and ten replicates do not replace domain experts. Assistance is one shot to one worker. The task set is seven items. Counseling, tax, tutoring, and operations research in particular need human review.
A few claims sit on thin ice even on the paper's own numbers. ρ=0.48 with p=0.187 and nine models cannot support a strong "these are weakly related capabilities" headline. The safer reading is that the two rankings fail to line up in this sample. Because the protocol withholds solution content, "assistance does not help" applies to this scaffold, not to drafting or correction. The worker is a 2023-class GPT-3.5-Turbo against 2026 assistants, so overload and execution drift are more likely. Automation is tool-free, which may under-rank models that shine in agent loops. Judge agreement is 71% and lower in augmentation, so ranks are sensitive to the judge panel.