How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review
Ming Li, Chenguang Wang, Xirui Li, Xinyue Zeng, Dianqi Li, Peng Shi, Dawei Zhou, Tianyi Zhou
cs.CL, cs.AI
2026-08-10
A controlled study rewriting 4,200 ICLR manuscripts shows AI reviewers can be swayed up to 0.93 points by rhetoric alone, mainly through evidence framing and novelty stance.
AI reviewers are now on both sides of the same pipeline. Some venues already use LLMs to draft preliminary reviews, and authors increasingly use LLMs to polish manuscripts before submission. When both happen at once, it's unclear whether review scores still track scientific merit or start tracking writing style instead. Prior work had shown that presentation-only edits can nudge AI-review scores upward, but nobody had isolated which rhetorical choices actually move the needle, or how reliable that effect is across models and conditions. A team from the University of Maryland, Virginia Tech, and MBZUAI ran a controlled experiment to answer this.
The team started from 120 anonymized ICLR 2026 submissions and built a corpus of 4,200 full manuscripts. The key design move was separating rhetoric from content: two rewriter agents, GPT-5.5 via Codex CLI and Opus 4.8 via Claude Code, rewrote each paper along six rhetorical dimensions in both directions: novelty stance, scope framing, evidence framing, contribution salience, technical register, and linguistic complexity. Rewrites touched the full manuscript, including prose, captions, and how results are presented, under a soft constraint that methods, experimental settings, and reported numbers stay unchanged. A programmatic checker separately verified that citation keys, equations, and code blocks weren't altered. Five AI reviewer models (Gemini 3.5 Flash-Lite, Qwen 3.5 Flash, GPT-5 mini, GPT-5.5, Claude Sonnet 5) then scored every original and rewritten manuscript under both a standard and a strict review protocol, with paired comparisons against the matched original. Beyond single-dimension rewrites, the team also tested joint, recursive, and reviewer-guided rewriting. The full run produced 42,396 valid review records at a direct API cost of $29,165.89.
Sensitivity concentrates in two dimensions. Evidence framing and novelty stance produce the largest and most consistent score changes, with scope framing a weaker second tier; the other three dimensions barely move scores. A positive evidence-framing rewrite raises the overall assessment (OA) by up to 0.93 points, a negative novelty-stance rewrite lowers it by up to 0.73, and evidence framing shifts weak-accept probability by 13 percentage points on average. That ranking holds across human-rated quality tiers, but the direction of movement depends heavily on the AI reviewer's own starting score: papers it initially rated 1-3 gain an average of 1.05 points from a positive evidence rewrite, while papers it rated 8-10 actually lose 0.19.
Stacking all six positive-direction edits doesn't compound cleanly. Opus 4.8's joint rewrite gains +0.289 under the standard protocol and +0.463 under strict, but the same joint rewrite done by GPT-5.5 barely moves the needle (+0.021 standard). Reviewer-guided second passes don't reliably beat an unguided second pass, and gains shrink further with additional rounds.
Rewriter and reviewer play different roles. The rewriter mostly sets how far apart the positive and negative variants land, while the reviewer determines the size and even the sign of the resulting score change: Qwen 3.5 Flash and GPT-5 mini swing widest, GPT-5.5 barely moves, and Sonnet 5 often moves in the opposite direction from the rest. Switching to the strict protocol drags mean OA down from 6.296 to 4.934, with 95% of papers scoring lower, though paper rankings stay largely stable (Spearman 0.862) and the protocol doesn't consistently amplify or dampen rhetorical sensitivity.
If a review pipeline has AI on both ends, style is not a neutral wrapper. Rewriting evidence framing and novelty stance can push a paper from a weak reject toward a weak accept without touching a single experimental number. For venues piloting AI-assisted review, average accuracy against human scores isn't the right benchmark alone; robustness to rhetorical rewriting matters, and that robustness varies by reviewer model and protocol in ways that don't transfer cleanly. For authors using LLMs to polish drafts, the same mechanism cuts both ways: whatever makes a rewrite persuasive to an AI reviewer is also a detectable signature that conference organizers could screen for.
The authors flag several caveats. The corpus comes from a single venue, ICLR 2026, so generalization to other conferences or fields is untested. Full-manuscript rewriting and multi-model review are expensive, and some planned reviews failed to produce valid structured output; that missingness may not be random, since some failures appear linked to topics that trigger model safety filters. The six rhetorical dimensions aren't orthogonal, so each rewrite is a whole-document intervention rather than an isolated linguistic variable. Most conditions use a single AI review per manuscript, with repeat-sampling stability checked only in a smaller audit. The results describe the specific models and prompts tested, not human reviewers or the full space of possible sampling variation.