Learned checklists cut LLM peer-review error 21-58%, beating NeurIPS official checklist
JessicaHullman · x · 2026-10-30
- In simulated LLM reviews of held-out NeurIPS 2021 & ICLR 2022 papers, the learned checklist (k=100) reduces error by 21-58%, outperforming the official NeurIPS 2021 checklist at every size.
- In a human study where reviewers rank NeurIPS papers, the selected checklist cuts ranking error by more than half.
- Key tension: a shared checklist applied by AI with high accuracy improves consistency, but correlates errors — reviewers may fixate on it over private info, so variance reduction must exceed added bias.
More from Research
- Commonroom beta: chat with neuron-steered LLM personas and hunt emergent personalities — repligate · 2026-10-30
- Elorian launches visual reasoning model Glimpse alongside DecoyCount benchmark — AndrewDai · 2026-10-30
- Instruction Capacity paper quantifies rule-following limits: GPT-4o mini holds ~37 rules at 95% accuracy — Hennadii Kutomanov · 2026-10-30
- Guidelabs launches Causal Diffusion Engine, adding interpretability to diffusion LLMs with no latency cost — juliusadml · 2026-10-30
- Alice & Bob says RSA-2048 falls in 3.14 days with 23,000 cat qubits — jedisct1 · 2026-10-30
- FlexGlue's 0.6 px average keypoint shift matches SIFT's subpixel refinement threshold — ducha_aiki · 2026-10-30