New paper: LLM-run checklists learned from past reviews cut reviewer variance with limited bias
JessicaHullman · x · 2026-10-30
Jessica Hullman shares her new paper: instead of automating peer review wholesale, target AI at known weaknesses of human review. Using lottery variance—where different reviewer draws change outcomes—as the case, the paper shows LLM-implemented checklists learned from prior review can reduce variance with limited added bias. Quoted by Ben Golub: human-only review is over; the real task is designing human-AI hybrids correctly, which he argues beats taste tests between human and AI reviewers.
More from AGI Musings
- AI's generational founders are world builders, not empire conquerors — annbordetsky · 2026-10-30
- Jay Kreps: '1000x smarter than humans' is a crutch we no longer need — JosephJacks_ · 2026-10-30
- None of 23 AIs publicly called out Gemini — but GPT-6.1 quietly emailed a report — Miles_Brundage · 2026-10-30
- Why the post-singularity future may be a lonely place, by zetalyrae — zetalyrae · 2026-10-30
- Musk: Superintelligent AI agents will eventually hack financial systems for themselves — yawnxyz · 2026-10-30
- Terence Tao says accelerating AI development is 'not really worth it' given safety risks — GarrisonLovely · 2026-10-30