NeurIPS Review Reflections: Compute Barriers and Score Calibration Issues
chhaviyadav_ · x · 2026-08-05
As the NeurIPS discussion period concludes, a researcher summarized key observations and pain points from this year's review process:
- Initial Meta-reviews: The new initiative was highly praised for helping authors prioritize rebuttal experiments and forcing reviewers/ACs to provide stronger justifications for rejections.
- Score Calibration: The shift to a 6-point scale caused noticeable adjustment issues. Reviewers hesitated to give a 5 (perceived as a 9 on a 10-point scale), restricting borderline papers to a 4.
- Compute & API Barriers: Reviewers continue to demand high-compute experiments at the 100B scale, indicating that securing compute and API credits is now an unavoidable hard threshold in academia.
- AI-generated Slop: Reviewers are actively giving low scores to papers with unclear writing and missing reproducibility details, often a byproduct of AI-generated text.
The author also expressed concern that unjust treatments during the review process might negatively impact reviewers' empathy and mindset in the future.
Related event: NeurIPS 2024 Review Mechanism Sparks Debate(2 posts)→
More from Research
- Daphne Koller's Essay: Drug Discovery Has No Magic Wands — AllThingsApx · 2026-08-05
- AI Scientist Summer Workshop Announces Agenda: Reliability and Closed-Loop Discovery — nc_frey · 2026-08-05
- Explained: Why Chain of Thought (CoT) Actually Works in LLMs — NaveenGRao · 2026-08-05
- Independently trained models develop 'universal' attention heads? New research shows predictable neuron scaling — CSProfKGD · 2026-08-05
- Connito Introduces Decentralized MoE Training Network on Bittensor — shibshib89 · 2026-08-05
- Improving Animal Welfare with Tech: Hyperspectral Imaging and E-Beam Vaccines in Poultry — NikoMcCarty · 2026-08-05