Blind test: agent papers beat humans on rigor but trail on novelty across 808 comparisons

SRSchmidgall · x · 2026-11-20

In 808 blinded full-text comparisons against 47,776 human arXiv preprints, agent papers scored higher on experimental rigor (+2.01) and measured claims (+2.08), while humans led on novelty (-0.39) and significance (-0.30). The overlay map shows 83% of the 936 agent studies cluster in the LLM 'peninsula' of the topic space, with nearest-neighbor distance only 0.61x that of humans — agents are rigorous but highly concentrated in topic choice.

Related event: Nearly 1,000 AI Agents Across 35 Labs Produce 936 Papers in Three Months(5 posts)→

Original post →

More from Research

Research channel →