Fact-Checking the NeurIPS-Pangram Story: v3.3.2 Flagged 14.9% of Human-Written Reviews as AI
tughanbulut · reddit · 2026-09-26
The author revisits the NeurIPS Position Track / Pangram controversy after decisions came out, correcting several claims from an earlier post against primary sources.
- The 79-paper tier wasn't "0.8 + solo author" but 0.8 plus multiple solo-authored submissions or another desk reject (NeurIPS blog, Table 5)
- The 22-paper tier also caught authors who left the AI declaration blank
- The "independent researcher" was one person, Sergey Berezin, who made no claims about the chairs' papers
- The viral 61% ESL false-positive figure came from 2023 tests of seven other detectors; Pangram's own report claims 0/89 false positives on those TOEFL essays, but that's unreplicated vendor data
What holds: Pangram's v4 report shows v3.3.2, the exact version NeurIPS used, flagged 14.9% (easy) and 4.5% (hard) of human-written, AI-polished peer reviews as fully "AI" — worse than both v3.0 and v4 — while the track policy explicitly allowed polishing. An ICML 2026 paper says per-window false-positive rates shouldn't be extrapolated to whole papers in either direction. The clearance status of the 123 conditional papers remains undisclosed.
More from Safety
- OpenAI Agent Breached Australia's Medicare Portal, Fueling the AI Doom Debate — Borthwick · 2026-09-26
- AI incidents are often human failures — worth more discussion than doomerism — basedjensen · 2026-09-26
- Oxford quietly let OpenAI train on the Bodleian Library's books — ross2000 · 2026-09-26
- skill-audit: Open-source pre-install auditor targets rising agent skill supply-chain risk — masiha97 · 2026-09-26
- Farmer's photo exposes 62 unpermitted gas generators powering Microsoft AI data center, $1.1M fine — mkheck · 2026-09-26
- Malware now asks commercial AI models to vote on its next attack move — TechNadu · 2026-09-26