FabScore Evaluates Fabrication in AI Papers
muhao_chen · x · 2026-07-11
FabScore for Evaluating Fabrication in AI Papers
This post introduces FabScore: a metric and method for fine-grained evaluation of "fabrication" in automated AI research. Accepted as a spotlight paper at the ICML 2026 AI for Science workshop and selected as one of the best papers, with the author noting an upcoming morning session presentation at COEX in Seoul.
Key Findings
- Evaluated 144 AI-generated papers from sources including Sakana AI's AI Scientist, MLR-Bench, Analemma AI's FARS, and the 2025 Agents4Science Open Conference.
- In 54 real conference submissions, about 70% contained at least one fabrication.
- Even among accepted papers, this proportion reached 59.3%.
Links to the paper and code are also provided.
More from Research
- Meta FAIR's Self-Direction turns agent failures into reusable LoRA skills, +13.8pp on Terminal-Bench — karen_ullrich · 2026-10-27
- DONIC hits 86.79% on bimanual dexterous tracking, crushing prior SOTA's 9.43% — liuziwei7 · 2026-10-27
- Why don't more LLMs ship 10M-token context? A researcher asks about attention tradeoffs — agarwl_ · 2026-10-27
- PSNR below 15 dB in papers likely means the metric was misused, researcher warns — kwangmoo_yi · 2026-10-27
- Ortet's first health eval: 34 open-weight models, 50 tasks, generic rankings fall short — thegautamkamath · 2026-10-27
- Scaling-fit discrepancy: output-head FLOPs are ~65% of compute at the smallest rung, ~3% at 535B — dlwh · 2026-10-27