Experts Rise Where LLMs Disagree: Using Cross-Model Disagreement to Target Expert Effort in LLM Codebook Revision for Large-Scale Annotation
Zeyu He, Zhuqian Zhou, Kirk Vanacore, Rene F. Kizilcec, Ting-Hao 'Kenneth' Huang
cs.CL, cs.AI, cs.HC, cs.LG
2026-09-23
Six LLMs flag codebook holes via disagreement. Experts label 120 hard tutoring turns with rationales; revised codebook hits 64.9% vs 57.8% after six months of expert revision.
Large-scale text annotation usually starts with a codebook: experts write down labels, examples, and decision rules, then humans or models apply them. Writing a codebook that actually works takes months. Literature scholars spent two semesters meeting weekly to code "acts of God" in fiction. A COVID-paper project had a biomedical expert hunt corner cases by hand. CAPC-CG scaled policy-directive labels to 3.3 million Chinese government paragraphs; three experts spent three months labeling about 500 paragraphs a week, comparing disagreements, and rewriting rules.
Penn State and Cornell tested a shortcut inside a live tutoring-transcript project. Two veteran math educators spent six months and roughly 300 hours each building a 29-move taxonomy (plus None, 30 classes) and reaching consensus on 4,032 tutor utterances. An early codebook already existed. The remaining months of expert time might be spent more tightly, by pointing experts at the cases where models already fight.
The working bet is that strong disagreement across LLMs flags codebook problems: ambiguity, a missing rule, or a hard boundary.
Three steps.
RL samples 10 utterances per target label with a greedy rule that covers unseen competing labels, 120 cases total. Experts never see the target label or any model vote. QA questions cover core behavior, necessary conditions, exclusions, boundaries, scope, and edge cases. CV edits five fields: Explanation, Examples, Near-Hits, Near-Misses, Non-Examples.
Evaluation uses a held-out set of 110 sessions and 4,032 expert-consensus labels. The same six models annotate it; Crowd Kit aggregates with eight methods (majority vote, Dawid-Skene, MACE, and others). Reported scores are the arithmetic mean across aggregators. Revision data and evaluation data do not overlap. Transcripts come from Eedi, UPChieve, and Third Space Learning, covering Grades 3-12 math tutoring in the US and UK.
Having experts explain hard cases beats having them rewrite the codebook themselves.
| Codebook | Labels eligible | Accuracy | Weighted F1 |
| Initial | 0 | 53.4% | 0.536 |
| Fully LLM-revised | 30 | 53.1% | 0.547 |
| CV (expert + LLM rest) | 30 | 53.7% | 0.557 |
| Best QA (ambiguity patterns + cross-label) | 25 | 60.5% | 0.613 |
| RL | 24 | 64.9% | 0.658 |
| Expert final (six months) | 30 | 57.8% | 0.583 |
RL beats the final codebook by 7.1 points. Paired t-tests are significant under all eight aggregators. On the 10 highest-disagreement labels (1,141 utterances), weighted F1 rises from 0.295 (initial) to 0.568 (RL); the expert final only reaches 0.429. Low-disagreement labels were already easier; RL still leads at 0.727.
Time tells the same story. CV takes 53 minutes per expert and barely moves accuracy; one variant drops to 52.0%. QA takes 3.0 hours and peaks at 60.5%. RL takes 2.8 hours, 120 labeled cases, about 1,658 rationale words, and gains 11.5 points over the initial codebook. The 300-hour estimate covers the whole original project, so it is not a matched cost, but the gap is two orders of magnitude.
Automatic prompt optimization without gold labels does not help. On a 586-utterance preliminary set, SPO scored 48.7% and APO 51.2%, both below the initial codebook at 55.0%.
Disagreement tracks difficulty. On 825 early expert-labeled utterances, majority-vote accuracy is 66.2% when three models agree (408 items), 36.0% when two agree (308), and 12.8% when all three split (109). Expert-persona prompting barely diversified votes; switching vendors did.
The two experts rated QA 7/7 for mental effort and RL 3.5/7. QA also scored 7/7 on surfacing distinctions they had not considered. If accuracy is the goal, pick RL. If the byproduct of hard questions is useful for later codebook work, QA is the other option.
For teams already using LLMs as annotators, the practical recipe is narrow: do not start by having experts edit prompts, and do not expect automatic prompt optimizers to invent missing expert rules. Run several models on the early codebook, pull high-entropy items, collect labels plus rationales, and let a model turn those rationales into reusable instructions.
This saves expert calendar time, not compute. Six models labeling 6,595 then 4,032 utterances is not cheap. What you get back is a human-readable codebook that can travel across data and models, not per-item alignment to one model. CV's failure is informative: editing LLM-written codebook text is as unstable as "prompting in the dark."
The gain is incremental. It is incremental on a real project with real experts and a six-month human baseline, not on a toy classifier.
The authors list the obvious ones. One tutoring project, two experts. The label inventory barely changed, so the workflow is untested when categories are added, dropped, or restructured. Carryover across the three workflows cannot be fully removed. The transcripts contain swapped speaker IDs, bad segmentation, and ASR errors, none of which were quantified. For the 6.1% of utterances with multiple consensus labels, evaluation keeps only the first.
A few more caveats sit in the design. Experts entered the study after finishing the six-month project, so "days instead of months" measures revision efficiency, not learning a domain from scratch. The final codebook was written for humans; the RL codebook was written for models. Higher LLM accuracy does not mean a human annotator would prefer the RL book. The reviser is locked to GPT 5.6 Sol; there is no ablation on that choice. 64.9% on a 30-way task is still low enough that production pipelines would want a second quality pass.