FabScore Evaluates Fabrication in AI Papers
muhao_chen · x · 2026-07-11
FabScore for Evaluating Fabrication in AI Papers
This post introduces FabScore: a metric and method for fine-grained evaluation of "fabrication" in automated AI research. Accepted as a spotlight paper at the ICML 2026 AI for Science workshop and selected as one of the best papers, with the author noting an upcoming morning session presentation at COEX in Seoul.
Key Findings
- Evaluated 144 AI-generated papers from sources including Sakana AI's AI Scientist, MLR-Bench, Analemma AI's FARS, and the 2025 Agents4Science Open Conference.
- In 54 real conference submissions, about 70% contained at least one fabrication.
- Even among accepted papers, this proportion reached 59.3%.
Links to the paper and code are also provided.
More from Research
- Proto-tools, a universal infrastructure layer for generative biology, now open source — jmschreiber91 · 2026-10-26
- New paper finds AI agents negotiate badly under uncertainty and break deals on purpose — xuanalogue · 2026-10-26
- MCP fault injection study: agent task scores drop 60% under realistic tool failures, scale doesn't help — dair_ai · 2026-10-26
- Study: AI reviewing AI misses most reward hacking; best models miss 25%+ of hacks — ronbodkin · 2026-10-26
- FLUX.3 Klein step-distilled to 4-8 steps with CFG baked in — Fit-Structure-6489 · 2026-10-26
- NLI alone rejects 5/6 new agent memories; a 0.45 similarity gate fixes it all — Alternative_Pin9598 · 2026-10-26