JHU paper: treating demographic annotation disagreement as the target for valid LLM annotation
krisgligoric · x · 2026-11-07
A Johns Hopkins paper (AACL 2026), "Multi-Perspective LLM Annotations for Valid Analyses in Subjective Tasks," introduces Perspective-Driven Inference: instead of assuming a single ground truth, it treats the distribution of annotations across demographic groups as the quantity of interest, estimated with a small human annotation budget. An adaptive sampling strategy concentrates human effort on groups where LLM proxies are least accurate. Evaluated on politeness and offensiveness rating, it shows targeted gains over uniform sampling while maintaining coverage.
More from Research
- Turn gaps average ~200ms: duplex voice models must plan replies mid-listen — smolix · 2026-11-07
- Why Higgs Realtime Duplex uses a 160ms step: it matches human decision speed — smolix · 2026-11-07
- ByteDance experiments suggest Full Bandwidth Transformer matches models twice its size — DimitrisPapail · 2026-11-07
- Ai2 brings 10+ robotics papers to CoRL 2026, spotlighting MolmoAct2 and MolmoBOT — allen_ai · 2026-11-07
- Applied Compute shares a production RL recipe for continual learning from live traffic — ypatil125 · 2026-11-07
- Anthropic Science Blog: AI Begins to Write, Fit and Revise Scientific Models — AnthropicAI · 2026-11-07