Yoav Artzi simulates 4 capped-reviewing mechanisms on real COLM data: 12-24% of accepted papers left out
yoavartzi · x · 2026-10-27
Cornell professor and COLM board member Yoav Artzi proposes four mechanisms for deciding which submissions get reviewed when conference reviewing capacity is capped, and simulates them on real COLM data (2,940 submissions, 856 accepted, capacity 1,500, max 5 papers per author) over 2,000 runs.
Framework: capacity split between a community set (submissions with an author who is a past accepted-paper author or current AC) and a randomly allocated set; abstracts due 2-3 weeks before the full deadline with prompt admission decisions, a side effect being fewer rushed submissions.
Mechanisms: P0 two random pools; P1 deterministic credits for past authorship; P2 queued admission based on past authorship; P3 queued admission with credits per year of past authorship.
Findings: under all mechanisms, 12%-24% of papers accepted in reality would be excluded from review (vs 49%-50% under a lottery); community papers get 76% admission odds vs 28%-36% for non-community papers; accepted papers become 79%-82% community papers (vs 61% in reality). Serving as an AC notably boosts admission odds for papers whose only community credential is the AC role (e.g. 75%→81% under P0).
Artzi stresses this doesn't represent COLM policy; it's posted to solicit community feedback.
More from Research
- Map of secure program synthesis: ruby-lean in Lean and the race to formally verify everything — burny_tech · 2026-10-27
- OpenAI claims rigorous Yang-Mills existence proof; physicist Woit calls the announcement misleadingly vague — burny_tech · 2026-10-27
- Researcher corrects 'Uncheatable forecast': admits flop-uncontrolled compute estimate — dlwh · 2026-10-27
- Buzz grows over claimed subexponential 3SAT and what it would mean for NP problems — burny_tech · 2026-10-27
- OpenAI internal model claims to refute ETH; Impagliazzo, Paturi and Williams to organize independent review — aminkarbasi · 2026-10-27
- D3-Gym: first auto-constructed verifiable environments for data-driven scientific discovery, shown at EMNLP 2026 — hhsun1 · 2026-10-27