~20% of ICLR reviews may be AI-generated: Heng Li pushes back, expert says AI review can beat average humans
anshulkundaje · x · 2026-10-28
Bioinformatician Heng Li (lh3) published a blog opposing journals' bans on fully AI-generated peer reviews, sparking debate.
Key facts
- A Pangram report claimed 21% of ICLR 2026 reviews are fully AI-generated; an independent study by Shen and Wang estimated 20% of ICLR reviews and 12% of Nature Communications reviews are mostly AI-written.
- Li cites three firsthand suspected cases: two manuscripts from his group likely reviewed by AI, and one paper he reviewed likely received fully AI comments from another reviewer.
- He distinguishes "AI-assisted" review (human drafts first, AI improves — acceptable) from "AI-generated" review (AI generates before careful reading; even later edits count, since AI already shaped the judgment).
- His core concern: AI lacks taste rather than knowledge, often missing the big picture when researchers must choose among plausible but conflicting options.
Counterpoint
- Stanford professor Anshul Kundaje disagrees with banning: in esoteric or cross-disciplinary areas, AI reviews may on average beat those of typical human reviewers; but he agrees the best practice is to read the paper and write your own review first, then augment with AI.
More from AGI Musings
- Rob LeClerc on why SWEs survive the AI era: 'it's the Red Queen effect' — robleclerc · 2026-10-28
- Geoffrey Irving: Public companies are allowed to choose not to risk killing everyone — geoffreyirving · 2026-10-28
- AI's most superhuman ability may be its capacity to forget — and keep secrets — corbtt · 2026-10-28
- Security researchers weigh whether AI-driven economic disruption will fuel global cybercrime — joshua_saxe · 2026-10-28
- Noah Smith: AI is the second-to-last important technological advance of the human age — abhiadesai · 2026-10-28
- When tasks can only be verified in the real world, RL rewards become reality itself — yacineMTB · 2026-10-28