GPT-6 and Claude pull 62% of review weaknesses from the appendix; humans, 8.4%

AI Reviews Are Biased Towards the Appendix

Saptarshi Ghosh, Mohammad Saim, Tianyu Jiang

cs.CL

2026-11-13

Four GPT-6 and Claude models tied 62% of review weaknesses to the appendix on 800 conference papers, against 8.4% in human ICLR reviews from 2021-2024; humans reached 27.6% in 2026.

What problem this solves

Reviewers already ask LLMs to list weaknesses, then carry those lists into conference reviews. At ICLR 2026, about one fifth of reviews were flagged as fully AI-generated. Earlier studies ask whether a model review agrees with humans on the decision. They rarely ask which page the criticism comes from.

The appendix sits after the references and usually has no page cap. Proofs, extra experiments, hyperparameters, and edge-case tables land there. That text is longer and less edited than the main body. A model pushed to be specific can mine it easily. Once the resulting list is treated as proof that the paper is weak, the core idea gets less attention.

Method

The Cincinnati study uses two slices. The main slice takes 200 papers each from ACL 2026, EMNLP 2026, COLM 2026, and ICLR 2027, restricted to appendices of at least 15 pages, 800 papers in total. ICLR 2027 papers are anonymous submissions. The other three venues contribute camera-ready copies, with author names, affiliations, emails, and repository links stripped, and with web search disabled. Human reviews cover ICLR 2021 through 2027, 200 papers a year. The 2027 papers overlap the main slice, and the union is about 2,000 papers.

The reviewers are GPT-6-Astra, GPT-6.1-Sol, Claude-Opus-5.5, and Claude-Sonnet-5.5, with GPT-5 as the prior generation. The prompt is a single sentence saying the user is reviewing for that conference and wants a conference-style review. Keeping it short stops format instructions from steering what gets criticized. The same model then maps each weakness to at most two sections, or marks it not specified.

The headline count is wide. Any weakness with some evidence in the appendix, whether the appendix alone or the appendix plus the main text, counts as appendix-related. Appendix-only means the weakness never touches the main text. Figures above one half, including the 62% mean, use the wide count.

For ICLR 2021-2024 the appendix minimum falls to 10 pages, since very long appendices were less common. One review is drawn at random from each paper, and GPT-6.1-Sol assigns sections. Those papers and reviews have been public for years and may already be in the training set, so the models do not re-review them. They are compared with LLM reviews of ICLR 2027.

A separate run gives GPT-6-Astra 100 ICLR 2027 papers through Codex, the PDF API, an API fed extracted plain text, and the web chat. Codex does not receive the PDF directly. It runs shell commands, and tool output is capped near 10k tokens, keeping the start and the end when the output is longer.

Results

Across the four venues, the new models tie about half or more of their weaknesses to appendix material. On ICLR 2027 the four-model mean is 62.00% ± 7.70 appendix-related and 14.19% ± 3.32 appendix-only. Across the four conferences, appendix-only is 17.33% for GPT and 10.16% for Claude.

SourceAppendix-relatedAppendix only
Four-model mean, ICLR 202762.00% ± 7.7014.19% ± 3.32
Humans, ICLR 2021-20248.40% ± 1.444.36% ± 1.03

GPT-5 falls between 17.9% and 29.6% on the four venues, every value under 30%. GPT-6-Astra and GPT-6.1-Sol land around 60%-70%, peaking at 72.4%.

All four access paths on the 100-paper slice stay above 50%. Under Codex, 72.4% of weaknesses are appendix-related and 27.6% come from the main body. Sending the full paper through the API in one call still favors the appendix, so the Codex cutoff does not explain the pattern.

Of 800 human reviews from ICLR 2021-2024, only 159 contain any appendix-related weakness. All 200 LLM reviews of ICLR 2027 contain at least one. The human rate stays near 6%-10% in review years 2020-2024, moves to 11.27% in 2025, and reaches 27.60% in 2026. The two increases follow GPT-5 in August 2025, then GPT-6-Astra in July 2026 and Claude-Opus-5.5 in September 2026.

Review yearConferenceAppendix-related
2020ICLR 202110.47%
2021ICLR 20228.37%
2022ICLR 20238.39%
2023ICLR 20246.37%
2024ICLR 20256.06%
2025ICLR 202611.27%
2026ICLR 202727.60%

Human weaknesses cluster on the first five pages, which carry the problem, the motivation, and the method. The models rarely challenge those pages. They draw more from pages 6 and 7, where results sit, and they still find targets past page 30. Human appendix comments fade as the pages go on.

The subject matter splits as well. Presentation, cases where key explanation is pushed into the appendix and the main text is not self-contained, accounts for 32.2% of human appendix weaknesses and 2.1% of LLM ones. LLMs assign 54.0% to methodology, 25.2% to analysis, and 13.6% to results, against 35.7%, 13.5%, and 6.5% for humans. Models match a main-text equation to an implementation detail in the appendix and ask for more seeds, significance tests, and confidence intervals. Those comments are often locally right, and they rarely reach the contribution.

Why it matters

What this study hands over is a measurement. A reviewer who reads the model list as the order of what is wrong will give a missing confidence interval about the same space as the core argument. An author who runs the same check before submission gets pushed toward edge-case experiments.

The practical use is narrow: let the model check details, and do not let it decide which objections are the main ones. An error in the appendix still needs a fix. The open question is weight. GPT-5 stays under 30%, the next generation clears 60%, and the API, the coding agent, and the web chat agree, so the skew sits in the model. The paper also tells authors not to delete appendix material in response. Reproducibility still depends on it.

Limitations

Section mapping is itself an LLM step. A model already inclined toward the appendix is asked where each weakness points. Vague items, and items that span sections, will be mistagged. The team checked a sample by hand and reported neither agreement nor an error rate.

The 62% and the 8.4% come from different years and different papers. Human appendices were allowed from 10 pages; the model sample starts at 15. A long appendix offers more lines to criticize, which lifts the percentage. Ordinary papers with short appendices were not included.

Under the wide rule, a weakness that cites both the main text and the appendix is counted entirely as appendix-related. Appendix-only is 14.19% versus 4.36%. A gap remains, and it is much smaller than 62% versus 8.4%. Both numbers belong in the same reading.

The move to 27.60% coincides with model release dates and is not a causal result. About one fifth of ICLR 2026 reviews were flagged as fully AI-generated, and those reviews remain in the human series. Some of the later rise may be AI text posted as reviews. The series does not show that human reviewers themselves began mining appendices more deeply.

Correctness, usefulness, and any effect on scores are unmeasured. How many appendix comments catch real mistakes is unknown. The study does not justify ignoring the appendix.

Terms

Source

Related papers

All paper explainers