RCT with 24,000 papers finds LLM-use policies barely affect peer review outcomes
windx0303 · x · 2026-11-12
Researchers from Microsoft Research, UPenn, Google Research and others ran a large-scale RCT and anonymous survey at ICML 2026 on how LLM policies affect peer review.
- Design: 24,000 papers and 17,000 reviewers randomized between a conservative policy (no LLM use) and a permissive policy (limited assistance).
- Findings: near-zero effects on paper decisions, scores, and reviewer confidence; reviews under the permissive policy were 5.5-7% longer.
- Noncompliance: survey (N=1,486) showed 22.5% of conservative-policy reviewers used an LLM anyway and 36.5% of permissive-policy reviewers reported a disallowed use.
- Implication: noncompliance alone doesn't explain the null results, as groups didn't differ meaningfully in LLM use. Full paper: arXiv 2609.19420.
More from Safety
- Five IndiaAI Mission-empanelled cloud providers lack data centres, barred from compute tender — kingslyj · 2026-11-12
- MCP server behaved for 3 tool calls, then rewrote tool descriptions to hunt credentials — ProfessionalLow8814 · 2026-11-12
- AWS shows fine-grained access control for agentic RAG on Bedrock knowledge bases — AWS ML Blog · 2026-11-12
- HF staffer clarifies: uncensored/abliterated models stay, only cyberattack models get pulled — mervenoyann · 2026-11-12
- Think tank funded by Anthropic, OpenAI and Google urges UK to loosen copyright for AI training — TuhinChakr · 2026-11-12
- Economist on AI plagiarism dispute: an innocent author would publish their Google Doc history — paulnovosad · 2026-11-12