PaperCritic: Training AI Reviewers to Assess the Soundness of Papers
Hyun Ryu, Ji Yong Cho, Eunho Yang, Moontae Lee, June Yong Yang, Sean Welleck, Carolyn Rose
cs.CL
2026-11-13
Trains a 4B reviewer with RL on an F1 reward over per-point precision and reference recall; 45.1 F1 on ICLR review beats Claude Opus 5 and transfers to code review.
AI conference submissions keep growing and the pool of qualified reviewers does not. LLM reviewers are the obvious fallback, but nearly all prior training recipes solve the wrong problem: generate a holistic review plus an overall score, then optimize similarity or preference against human reviews. That is human-review imitation, and the target is noisy. Reviewing serves several purposes at once (TMLR lists soundness, interestingness, and clarity), the latter two largely subjective, and human reviews vary wildly in quality. PaperSound-23K quantifies this: of 23K human- and LLM-written review points, 45% rest on incorrect premises, and 81.1% of the unsound ones were written by humans. Imitating that signal teaches a model to be confidently wrong.
The paper narrows the job to the most objective criterion: soundness, whether a paper's claims are backed by accurate and convincing evidence.
The criteria come from a human study first. Seven AI researchers (three-plus years of experience each) scored review points and explained their judgments; interviews distilled two universal criteria. A review point is high-quality if and only if it is sound (its premises hold according to the paper or field knowledge) and substantive (it raises a moderate-to-significant concern that a main claim or its central evidence may not hold).
A review is then treated as a list of points with two objectives:
PaperSound-23K backs this with 989 ICLR 2020-2023 submissions and 23,215 review points (62.4% human, 37.6% LLM-generated), quality-labeled by a GPT-5.2/5.4 pipeline. Human-model agreement reaches 0.624 versus 0.597 for human-human, so the automated labels hold up. Only 27.1% of points clear both bars.
Training runs in two stages. SFT uses only sound and substantive reference points, interleaved across target claims to push coverage first. Online RL (GRPO) then optimizes a review-level F1 reward: precision is estimated by two reward models (a GPT-5.4-distilled 4B for soundness; Qwen3.6-35B-A3B for substantiveness), and recall by a matcher against the reference set. One counterintuitive ablation: recursively generating one point at a time with point-wise rewards scores 37.1 F1, well below 45.1 for one-shot generation with the review-level F1.
On the PaperSound-23K test set (100 papers, GPT-5.4 as grader):
| Model | Precision | Recall | F1 |
| Qwen3-4B base | 11.3 | 29.6 | 11.2 |
| SFT | 23.0 | 25.6 | 16.6 |
| +RL, ROUGE similarity reward | 24.9 | 34.2 | 23.7 |
| +DPO | 24.8 | 35.2 | 23.7 |
| PaperCritic-4B | 47.4 | 51.4 | 45.1 |
| Claude Opus 5 | 29.3 | 73.8 | 39.1 |
| DeepSeek-V4-Flash | 36.0 | 55.2 | 39.3 |
That is 21.4 points over the best conventionally trained 4B baseline, 25.9 over Qwen3-235B in thinking mode, and 5.8/6.0 over the two strongest frontier models. The frontier failure mode is consistent: high recall, low precision. They cover the reference concerns but pad them with unsupported claims. In a 7-expert preference study, PaperCritic-4B wins 5 of 7 head-to-head comparisons against DeepSeek-V4-Flash and supplies 19 of the 35 top-5 selected points.
Generalization is the second surprise. Trained only on ICLR submissions, the model climbs from 22.7 to 58.7 F1 on PeerReview Bench (Nature-family papers, a 2.6x gain), 7.2 to 10.5 on SWE-PRBench (code review), 13.8 to 20.1 on CLAUSE-Legal (contract review), and 78.0 to 90.0 recall on StoryFeedback. The SFT stage hurts on all four; RL carries the gains.
The real contribution is the objective function, not the checkpoint. Per-point evaluation with an F1 reward turns reviewing from fuzzy imitation into a well-defined optimization problem, and the same recipe fits any task whose output is a list of findings; the four OOD results are the evidence. Two practical takeaways: use it (or something like it) to pre-screen your own submissions for soundness holes before submitting, and do not trust frontier models as reviewers out of the box. Opus 5's precision of 29.3 means most of its listed issues would not survive scrutiny; handing that review to authors would actively mislead them.
The authors list three: scope is soundness only (no novelty, clarity, or accept/reject recommendation; an assistant, not a replacement); evaluation is closed-book, excluding critiques that require comparing against external literature, which real reviewing often demands; and recall is reference-bound, so an unmatched point is not necessarily wrong, only uncovered.
Two more from the numbers. The 5.8-point edge over DeepSeek-V4-Flash has a paired bootstrap CI of [-0.75, +12.62], which includes zero; only the gaps against Opus, Gemini, and Sonnet are statistically resolved. And both the training labels and the main evaluation come from the GPT-5 family. Re-grading with Gemini 3.8 Flash preserves the ordering (44.4 versus 33.3/33.1), but the definition of what counts as sound still rests on one model family's judgment, with human-model agreement of 0.624, comparable to human-human yet hardly high. The 8B variant drops to 34.9 F1, attributed to base-model differences and insufficient tuning, and RL induces formulaic phrasing ('can you' openers, 'it is unclear that') whose cause the paper leaves open.