Princeton's PaperScorer fine-tunes Qwen to predict paper acceptance, hitting 77.5% accuracy
IanArawjo · x · 2026-10-27
Princeton researchers Sachin Konan, Jonathan Liu, and Zhuang Liu released PaperScorer, a NeurIPS '26 project that predicts paper acceptance from anonymized submissions, plus a web service: upload an anonymized PDF and receive a calibrated probability labeled Leans Reject, Leans Accept, or Likely Accept.
Method:
- PaperScorer-Vision: fine-tuned Qwen2.5-VL-7B reading screenshots, layout, and figures
- PaperScorer-Text: fine-tuned Qwen2.5-7B on text-only markdown
- Output is a calibrated acceptance chance, not a review
Data and results:
- Two fully anonymized corpora: 24,888 ICLR papers from OpenReview ('20–'26) and 83,656 arXiv papers (NeurIPS/ICML/CVPR etc.)
- 68.3% accuracy on the ICLR test set and 77.5% on arXiv, beating vision variants of Gemini-3.1-Pro and GPT-5.5
- Notably strong on the tails of highly rated papers
Notable validation cases: Mamba, rejected at ICLR 2024, gets a 71% score (2.3× the 31% ICLR base rate) — arguably it deserved acceptance; the early version of A Decade's Battle on Dataset Bias is correctly predicted as a reject, while its later accepted ICLR version is correctly predicted as an accept. Two of the showcased well-known good papers were actually rejected.
Limits: 5 papers per mailbox per day, first 15 pages only, results may be delayed by queue load.
More from Models
- Kimi Team proposes Attention Residuals, replacing fixed residual accumulation with learned attention — burny_tech · 2026-10-27
- OpenAI claims Exponential Time Hypothesis is false, solving 3SAT in subexponential time — fortnow · 2026-10-27
- New benchmark's outlier forces a custom x-axis as Claude Opus 5.5 tops test-time scaling — andrew_n_carr · 2026-10-27
- Two open checkpoints show a full JEPA-pretrain-to-GRPO RL recipe on small LLMs — burny_tech · 2026-10-27
- Google AI Overview labels game a 'survival game' then corrects itself in the same answer — Rare_Bat2584 · 2026-10-27
- Anthropic quietly updates Claude Fable page, hinting Fable 5.5 launch is imminent — imjustnewatai · 2026-10-27