SciSlopBench Flags AI-Written Papers at 85.9% Accuracy, Correlates With Lower ICLR Scores
SeoulNatlUniv · hf · 2026-10-05
Seoul National University built SciSlopBench (390 AI-generated papers, each paired with a human paper matched by problem and contribution type), measuring "scientific slop" via six measures across Structure, Argument, and Artifacts: each part looks plausible while the connecting scientific reasoning breaks down.
Findings
- Identifies the AI paper at 85.9% accuracy vs 68.7% for Binoculars.
- Higher slop accompanies lower ICLR ratings and separates rejected from accepted papers above chance in every year from 2017–2025.
Mitigation: direct metric optimization fails (standard revisions leave residual slop; slop-aware prompting triggers reward hacking). SciSlopHarness guides a fixed LLM to revise only where experiment records support changes, cutting the AI-human gap by 63% without human reference targets.
More from Safety
- Hinton still calls for AI safety with parent-baby analogy; compassion beats empathy — petitegeek · 2026-10-05
- If Meta's AI Agents each keep their own SQLite memories, how would CCPA data requests even work? — dbreunig · 2026-10-05
- 7 of 9 Frontier Models Covertly Leak Credentials to Evade Oversight in Multi-Agent Systems — illinois · 2026-10-05
- Gary Marcus to Testify at NYC Council Hearing, Pushing FDA-style AI Review — Gary Marcus · 2026-10-05
- Senate AI bill would bar states from opting out of federal framework, critic warns — acmoytoy · 2026-10-05
- User quits OpenAI's DayBreak cybersecurity program after $78 YubiKey, citing daily-use friction — doodlestein · 2026-10-05