ICLR's shift from 10-point to 4-point scores inflates quantization noise 11x

valuat · reddit · 2026-11-05

Reacting to ICLR changing its review scale from 1-10 to 1-4, the author shares an (unprompted) Gemini-generated measurement-theory critique:

Quantization noise: coarse binning multiplies rounding-error variance 11x vs the 10-point scale; AC-averaged review scores inherit a much larger error component.
Lost nuance: reviewers avoid extreme anchors, collapsing the distribution into a coin toss; on a 10-point scale an enthusiastic 8 could lift a skeptical 5 into acceptance — the same champion now yields a 3.0 average indistinguishable from two lukewarm reviews.
Combinatorial floor: 3 reviewers on a 1-4 scale produce only 20 multisets and 10 possible means, so hundreds of papers pile up at identical ties like 2.67 or 3.00, making scores useless for ranking and leaving final calls to a single AC's taste.

The table contrasts what Program Chairs think they're fixing (calibration disputes, neutral cop-outs, score-sorting) with actual consequences (existential 2→3 swings, forced polarization, higher AC workload and variance). Conclusion: removing precision from a noisy instrument doesn't clean the signal — it forces unprincipled tie-breaks.

Original post →

ICLR's shift from 10-point to 4-point scores inflates quantization noise 11x

More from Fun