Does Multi-Agent Debate Improve AI Feedback on Research Papers?
Tomas Havranek, Zuzana Irsova
econ.GN, cs.CL, cs.MA
2026-07-16
Blind-ranked by 44 meta-analysis authors, a single-pass model beat two multi-agent debate tools by 0.66 rank points at 1/35 the cost; AI judges ranked real referee reports last.
Multi-agent debate, letting several AI agents each take a position and critique each other before synthesizing a final answer, is a popular recipe for more reliable output. Applied to writing peer-review-style feedback on research papers, does it actually beat one model reading the paper once and writing a report? The stakes are real: if debate-style review is clearly better, high-value tasks like refereeing and research feedback may be worth paying tens of times the inference cost. If it is not, a lot of agent orchestration may just be burning money.
Earlier comparisons of this kind either let an AI grade the AI, which is circular, or did not control report length and format, so longer reports won by sheer volume. Here the authors hand the verdict to the paper's own authors and lock every report to the same template and word budget, isolating content.
The authors recruited 55 economics meta-analyses (27 of their own, 28 external); 44 (80%) returned at least one ranking. Meta-analysis was chosen because the genre is standardized and its claims are checkable, so criticism can be judged against an objective reference.
The three AI review reports were:
All three were squeezed into one template and roughly 1,000 words, identity-masked, and shown in random order. The authors of each paper then blind-ranked them by usefulness for improving their own work. The hypotheses and analysis were pre-registered.
One piece of context: mad-research and paper-workshop were built by the authors, who expected them to win.
Authors put single pass first, clearly:
| Report | Mean rank (lower is better) | Ranked first |
| Single pass | 1.59 | 55% |
| paper-workshop | 2.16 | 27% |
| mad-research | 2.25 | 18% |
Single pass beat mad-research by 0.66 rank points (95% CI 0.32 to 1.00, Holm p=0.005) and paper-workshop by 0.57 (0.16 to 0.95, p=0.026). The two debate tools were indistinguishable from each other (0.09, p=0.75). The pre-registered main hypothesis, that at least one debate tool beats single pass, was rejected.
Cost tells the other half. The dollar ratio was about 1 : 9.2 : 35. paper-workshop cost 35 times single pass (about $7.20 against $0.20) and close to 30 times the tokens, for a worse author verdict.
The second finding is worse. Among the 21 papers whose authors still recalled their journal referee report, the human report was ranked first 71% of the time and never last. The three AI judges put the human report last on 75 of 78 judge-by-paper placements; on the authors' own 26 papers, Gemini ranked it last 100% of the time, Claude 96%, GPT 92%. AI judges were systematically devaluing expert human opinion.
The third is the Gemini reversal. Single pass and paper-workshop were both written by Claude and mad-research used several families, so Gemini, the one judge whose own model family wrote none of the reports, is the relatively neutral arbiter. Yet Gemini would have put the most expensive tool, paper-workshop, first (mean rank 1.45, first on 64%). Had Gemini ranked in the authors' place, it would have reversed their preference. Author-judge rank correlation was weak throughout: Gemini τ=0.14 (not distinguishable from zero), GPT 0.27, Claude 0.20.
For anyone building agent pipelines, this is a direct reminder. On the task of giving feedback on finished research papers, stacking more agents and burning more tokens did not help, and the loser here was not some strong baseline but the plainest possible read-and-write. A simple single pass should be the default starting point, and a debate or multi-agent workflow should prove it earns back that 35x cost before shipping.
For anyone using AI as a judge, the warning is sharper. AI judges correlated weakly with author preference and systematically put real referee reports last. On whether AI can substitute for a human academic referee, the evidence points to "not yet". Do not let one model's taste override the author's own judgment; the Gemini reversal is a ready-made counterexample.
The authors flag several. First, squeezing all reports to about 1,000 words may bias the contest toward single pass, since debate tools normally win on longer, denser output. Second, only economics meta-analyses were tested, a genre chosen precisely because its claims are checkable, which may be where debate adds the least. Third, paper-workshop ran in a light configuration; the full version was not tested. Fourth, the study measures perceived usefulness, not whether papers actually improved after the feedback.
Two more concerns stand out. Within-paper agreement was weak (mean pairwise τ=0.049), so the rankings are noisy and the 0.66-rank-point gap should be read with that in mind. The authors also note that participating authors may have used AI to help rank; weak concordance with the judges suggests they mostly judged for themselves, but that uncertainty cannot be fully ruled out.