FULL STORY

Crisis of Confidence in AI Peer Review

Academics are expressing strong dissatisfaction with AI conference peer reviews, citing perfunctory rebuttals, compromised double-blind processes by AI search agents, and accepted fake content.

2026-07-27 ~ 2026-08-05 · 4 episodes · 21 posts

Episode 1 · NeurIPS Review Season Sparks Debate Over AI Reviewers and Peer Review (2026-07-27, 2 posts)

The release of NeurIPS review results has sparked academic outrage over poor review quality, with reviewers ignoring appendix details. The use of AI in reviewing has become a central controversy, prompting renewed calls to abolish anonymous peer review.

Episode 2 · NeurIPS 2026 Rebuttal Deemed Perfunctory, Reviewer Apathy Sparks Outcry (2026-08-03, 6 posts)

After NeurIPS 2026 decisions were released, the academic community again erupted in criticism of top conference peer review. Multiple researchers pointed out that the rebuttal phase is perfunctory, with reviewers being dismissive, raising broad doubts about the effectiveness of the review process.

Confirmed

  • A NeurIPS paper author shared a hardcore rebuttal title, directly stating that after pointing out the reviewer had not read the paper initially, the reviewer still did not read it in the second round. This anecdote was spread by @RylanSchaeffer and @evijit, sparking discussion.
  • @miniapeur noted that reviewers (typically Reviewer 2) often leave a dismissive comment like "Thank you for your detailed response, I will maintain my score" in the last hour before the deadline.
  • @karimjerbineuro confirmed that the rebuttal phase has ended, joking that being ignored by an anonymous reviewer is more hurtful than being stood up in real life, while thanking reviewers who actively participated and area chairs (ACs) who remained constructive.
  • As final results came out, @miniapeur summarized that researchers complain their hard-written rebuttals often go unanswered and have no substantial impact on final acceptance.

Why it matters

  • @SoloGen and others emphasized that if conferences cannot even guarantee that reviewers respond to authors in a timely manner during the rebuttal phase, the entire peer review process's effectiveness is questionable. This is not just daily academic grumbling but reflects a structural crisis of out-of-control review quality and "going through the motions" in academic communication amid the explosive growth of top AI conferences.

Episode 3 · AI Conference Review Crisis: Low Response, Hostility, and AI-Fabricated Content Accepted (2026-08-03, 10 posts)

Recent multiple AI researchers have raised concerns about the peer review mechanisms at top conferences such as NeurIPS, ICML, ICLR, and EMNLP/ARR. They point out that with a surge in submissions and a severe imbalance in the supply of qualified reviewers, response rates are extremely low, review attitudes tend to be hostile, and even AI-generated fabricated content has been accepted. While the existing material is insufficient to conclude that the review system has completely collapsed, numerous specific cases indicate that current review quality and accountability mechanisms are under immense pressure, directly affecting the quality and credibility of academic selection.

Confirmed

  • RexDouglass and srchvrs note that AI paper submissions have surged (up to 10x), but the growth of qualified reviewers lags far behind (only 2-3x), with many new reviewers being less experienced PhD students. Under unpaid, high-pressure conditions, review quality decline is almost inevitable.
  • RexDouglass mentions that NeurIPS 2026 reviewers had a response rate below 5% even after two reminders; the EMNLP ARR system has been reported to have meta-reviewers making factual errors and repeatedly requesting unnecessary ablation studies for submissions clearly labeled as datasets.
  • a1zhang states that reviewers often request experiments already present in the paper and refuse to communicate or disappear during the rebuttal phase. RexDouglass also relays the pain point of authors investing significant time and API costs to run experiments only to face reviewer ghosting.
  • deepakns and anpaure believe that the review atmosphere at NeurIPS, ICML, and ICLR is hostile, making researchers irritable and suspecting a vicious cycle of "retaliatory scoring."
  • gabriberton notes that in ICLR 2026 reviews, they flagged 11 papers with AI-generated fake references or authors, but 2 were still accepted as oral presentations.

Unconfirmed

  • Claims of "retaliatory scoring" and "full pipeline takeover by LLMs" are based on personal observations and anecdotes; existing material is insufficient to assess their prevalence.
  • There is no official systematic statistics or formal response from conference organizers regarding reviewer ghosting and ineffective communication.

Why it matters

  • Peer review is the core mechanism for filtering research at top AI conferences. If low response rates, erroneous reviews, and AI-fabricated content coexist, it not only buries valuable research but also allows low-quality or distorted content into the main conference.
  • Many researchers therefore call for substantive reforms to the review system, including longer discussion periods, accountability and incentive mechanisms for reviewers, or reducing reliance on single-round scores.

Episode 4 · AI Agents Are Undermining Double-Blind Peer Review (2026-08-03, 3 posts)

The AI academic community is facing a double-blind review crisis, as advanced AI search agents make it easy for reviewers to de-anonymize submissions by matching them with arXiv preprints. This growing capability is effectively undermining the anonymity of top conference peer reviews.