Debate over AI slop papers: Leibo predicts they will form separable clusters

DeepMind researcher Joel Leibo (@joelbot3000) and @cedcolas engaged in a multi-round debate over whether AI-generated "slop" papers can be reliably detected, with the core disagreement centering on how subjective slop judgments are and whether testable empirical predictions can be made.

Confirmed

• Positions: Leibo argues that true slop has clear hallmarks—it sounds sophisticated but is hollow, doesn't lead anywhere relative to existing research, and doesn't genuinely connect to past literature—so identifying it isn't that subjective; he also acknowledges that half-hearted human-written papers exist too. cedcolas distinguishes between "AI-style writing" and "AI slop": detectors are feasible for the former, while the latter is inherently subjective and hard to detect reliably; he says he currently writes papers with AI and doesn't consider the results slop.
• Leibo's testable prediction: clustering the initial versions of all OpenReview submissions using LLMs from a year or two out, slop papers would visibly separate into their own cluster, distinguishable from LLM-assisted papers and sincere first-year PhD student papers.

Not yet confirmed

• The clustering prediction is itself a forecast about the next year or two, with no empirical validation yet; follow-up discussion also noted other confounding factors, so the conclusion remains to be tested.

Why it matters

• If the prediction holds, it would provide empirical evidence that AI slop papers are detectable, directly informing how review platforms like OpenReview handle AI-generated submissions; if not, it would support the view that slop judgments are highly subjective and hard to automate, with implications for how academic publishing defines AI misuse.

2026-09-25 ~ 2026-09-26 · 5 related posts

• [source] AI-shaped writing isn't AI slop: detectors will catch the former, the latter stays subjective — cedcolas · 2026-09-25
• Actual slop isn't that subjective: profound-sounding but trivial text is classifiable — joelbot3000 · 2026-09-26
• [source] LLM clustering of OpenReview papers will expose a separable AI-slop cluster, researcher predicts — joelbot3000 · 2026-09-26
• [source] LLM Clustering Could Separate ICLR Slop Papers Into Their Own Cluster, Devs Argue — joelbot3000 · 2026-09-26

1 near-duplicate retellings: joelbot3000