NeurIPS Reviewer Critiques 6-Point Scale and Unrealistic Compute Demands

chhaviyadav_ · x · 2026-08-05

Following the end of the NeurIPS discussion period, a reviewer shared six key observations about this year's review process:

Initial Meta-reviews: The new initiative was well-received, helping authors prioritize rebuttal experiments.
Score Calibration Issues: The transition to a 6-point scale caused confusion, with many borderline accepts (4) actually representing what would be an accept on a 10-point scale.
Compute Arms Race: Reviewers continue to demand extremely high-compute experiments (e.g., 100B scale), disadvantaging teams without massive compute and API credits.
Higher Standards for Position Papers: While submissions to this track are easier, acceptance standards have risen significantly, rejecting mere surveys.
Anti AI-Slop: Reviewers are giving low scores to poorly written, unclear submissions suspected of being AI-generated slop lacking reproducibility details.
Sentiment Impact: Unfair rejections might affect reviewers' future attitudes, making them either more considerate or harsher.

More from Research