LLM-Generated or Human-Written? Comparing Review and Non-Review Papers on ArXiv
Yanai Elazar, Maria Antoniak
EMNLP 2026
cs.DL, cs.CL, cs.CY
2026-01-20
Two detectors show CS reviews are more often LLM-written (Pangram 43.3% vs 23.3% in 2025), but non-review LLM papers outnumber them nearly 6 to 1.
On 31 October 2025, arXiv stopped accepting unpublished review, survey, and position papers in Computer Science. The stated reason was a surge of LLM-generated text in those genres, overloading volunteer moderators. The announcement cited no numbers.
This EMNLP 2026 paper turns that policy into two measurable questions. Do review papers actually contain more LLM-written text than ordinary research papers? And once the ban lands, which CS subfields lose a large share of their submissions?
The pipeline has two stages, and it deliberately uses titles and abstracts only. That matches how arXiv does the first pass: an automatic filter on title and abstract, with flagged papers sent to a human. The main analysis is aligned with what moderators see, not a shortcut.
Stage one labels papers as review or not. The two authors annotated 200 CS papers from titles and abstracts, 100 with review-like keywords and 100 with experimental keywords, at Cohen's κ = 0.85. On a validation split they compared Llama 3.3, Gemma 2, GPT-OSS, and GPT-4o-mini. Gemma 2 hit 95.6 F1 and runs locally, so they kept it. On a held-out test set of 140 papers it scores 92.9 precision, 91.2 recall, and 92.0 F1. GPT-4o-mini matches that F1 but is closed-source.
Stage two estimates LLM writing with two independent detectors.
The corpus is Cornell's official Kaggle arXiv dump, frozen on 15 December 2025. arxiv-domains covers CS, mathematics, statistics, and physics, up to 500 papers per month, 124,461 in total. cs-subcategories covers the ten largest CS categories, 138,244 papers. 2020-2022 calibrates the detectors; 2023-2025 is the analysis window.
Review papers in CS do look more LLM-written. Adjusted Alpha over the post-LLM cohort is 21.4% for reviews versus 14.0% for non-reviews. Both sides rise year by year, and the gap does not close.
| Detector | Type | 2023 | 2025 |
| Alpha | Review | 12.9% | 28.2% |
| Alpha | Non-review | 6.2% | 18.9% |
| Pangram | Review | 7.4% | 43.3% |
| Pangram | Non-review | 2.7% | 23.3% |
Raw counts reverse the policy rationale. Extrapolating Pangram to all CS papers in 2025 gives about 4,783 LLM-flagged reviews and 26,801 LLM-flagged non-reviews, nearly six times as many. Total CS volume that year is about 12.1k reviews versus 141.8k non-reviews. The pile on the moderators' desk is not the review pile.
The cost is also uneven across subfields. Computers & Society has a 49.2% review rate; software engineering 23.4%; computer vision 2.4%. A blanket ban could drop half of Computers & Society submissions and about 3% of CV. Across ten subcategories, review rate and AI-flag rate correlate at Pearson ρ = 0.886. Cryptography and Security shows the highest AI signal in reviews (Alpha 31.35%, Pangram 24.68%); NLP (cs.CL) the lowest (22.06% / 10.88%). OpenAlex topics tell the same story at a finer grain: Safety Research is 59.7% reviews, Education 33.8%, computational mechanics 2.6%. Topics currently arguing about AI's social effects are the ones the policy cuts deepest.
Full-text checks do not overturn the trend. Of 437 CS papers Pangram flagged from the abstract, 430 full texts were recovered; 57.9% still flagged, 216/365 non-reviews (59.2%) and 33/65 reviews (50.8%). Reviews more often show the AI signal only in the abstract. On full text, 2025 CS rates fall to about 26% for reviews and 16% for non-reviews, roughly half the abstract estimates, with the same upward year trend and the same review/non-review gap.
If the goal is to save moderator time, banning reviews aims at the smaller stack. About 85% of Pangram-flagged LLM papers in CS in 2025 are non-reviews. If the goal is quality control of generated text, non-reviews are rising too, from a base an order of magnitude larger.
For authors, the live issue is field asymmetry. Computers and society, HCI, safety, and education organize a lot of their debate in reviews and position papers; CV and machine learning barely notice. The policy is written as "all of CS." The bill lands on a few subcommunities.
For people who build detectors, the useful move is population-level estimation plus a pre-LLM false-positive calibration. Alpha's 13.1% review false-positive rate is a warning: leave it uncorrected and you systematically inflate the "reviews are dirtier" story.
This is a policy audit, not a new detector. Code is public. The two detectors disagree on levels (28.2% Alpha vs 43.3% Pangram for 2025 CS reviews). Trust the direction and the order of magnitude, not any single percentage point.
The authors list the hard problems. Alpha rides on adjective distributions; a new generator or heavy human editing can shift it. Rogan-Gladen also assumes the false-positive pattern does not move with real AI use. Pangram had zero pre-LLM false positives, but there is no ground truth after 2022, so nobody knows how much of 43.3% is overkill.
92.0 F1 on the review classifier is high, yet the label itself encodes a mainstream CS idea of what a "review" is. Tutorials and hybrid papers that mix experiments with a stance get squeezed into a gray zone. arXiv moderators likely share that idea, so the paper is simulating the policy, not offering a neutral taxonomy for every subfield.
The larger hole is the sample: posted papers only. Rejected, withdrawn, and never-cleared submissions are invisible. The authors asked arXiv for submission logs and did not get them. Reported rates are therefore closer to a lower bound. Detectors also cannot tell a fully generated manuscript from a non-native speaker running a model over their sentences. Mixing those in one "LLM paper" bucket will bias any policy debate.
A manual check of OpenAlex author links found 9 errors in 20 authors, mostly merged profiles. Results on seniority and country stay in the appendix; the main text does not lean on them. That restraint is correct.