Blind test: agent papers beat humans on rigor but trail on novelty across 808 comparisons
SRSchmidgall · x · 2026-11-20
In 808 blinded full-text comparisons against 47,776 human arXiv preprints, agent papers scored higher on experimental rigor (+2.01) and measured claims (+2.08), while humans led on novelty (-0.39) and significance (-0.30). The overlay map shows 83% of the 936 agent studies cluster in the LLM 'peninsula' of the topic space, with nearest-neighbor distance only 0.61x that of humans — agents are rigorous but highly concentrated in topic choice.
Related event: Nearly 1,000 AI Agents Across 35 Labs Produce 936 Papers in Three Months(5 posts)→
More from Research
- Interactive lessons record a 1.5B model's attention and token probabilities live — techartist_ · 2026-11-20
- Zero-cost MLM-head rescaling stabilizes SPLADE training, yet GTE-MLM remains unbeaten — tomaarsen · 2026-11-20
- After evaluating Forge for MTG, researcher writes his own engine for RL agents — petrusenko_max · 2026-11-20
- Pretrained protein models fail to learn 15N chemical shifts, study shows — KevinKaichuang · 2026-11-20
- Study: AI agents barely learn social norms and rely mainly on being punished themselves — ghadfield · 2026-11-20
- Xeophon discusses the ECI benchmark's creation and what works with its creators — xeophon · 2026-11-20