Scholar Calls Peer Review Metrics Superstition; LLMs That Actually Read Papers Spark Debate

On August 17, scholar Rex Douglass fired off a series of posts criticizing the research ecosystem: verifying paper content directly is prohibitively expensive, pushing many researchers toward proxy metrics like peer review and citation counts—practices he likened to outright superstition. He also argued that LLMs can now genuinely read papers the way a methodologist would and rapidly verify questions, yet the technology has been banned in some quarters for being seen as a replacement for traditional methods, igniting a methodological debate.

Confirmed

• Douglass's core criticism: proving that empirical research is internally coherent is extremely difficult and expensive, so many people lean on substitute indicators like peer review and citation counts, forming an irrational superstition instead of directly and carefully reading papers for substantive content
• His reasoning: inferring an artifact's value from metadata alone requires more information than directly consuming the artifact itself; when evaluating content becomes too costly, people invent superstitions in place of thinking
• He praised LLMs for reading papers like a methodologist (not a blind cheerleader), providing evidence-based question rankings that let people quickly verify questions and move forward
• Point of contention: some have banned the technology on the grounds that LLMs replace traditional methods

Unconfirmed

• The specific settings, scope, and groups involved in the "LLM ban"—the posts offer no details

Why it matters

• Some replies were pessimistic, arguing science has long been awash in mediocrities and ritualists, that ordinary people paying attention to science is itself a disaster, and that LLMs will only produce more "extremely confident tourists"
• The debate strikes at the root of research evaluation: by Douglass's logic, verification costs shaped the metadata-centric gatekeeping mechanism, and if LLMs drastically lower the cost of "actually reading papers," the legitimacy of this proxy-metric system will come under reconsideration

2026-08-17 ~ 2026-08-17 · 5 related posts

• People Invent Superstitions When Evaluating Information Costs Too Much — RexDouglass · 2026-08-17
• [source] Scholar Critiques Peer Review: Citations Are Superstition — RexDouglass · 2026-08-17
• Laziness makes religion inevitable: criticizing proxy metrics in science — RexDouglass · 2026-08-17
• [source] LLMs can read papers like methodologists, enabling quick verification — RexDouglass · 2026-08-17
• [source] LLMs Can Actually Read Papers, Sparking Methodological Debate — RexDouglass · 2026-08-17