NovGauge benchmark finds LLMs judge paper novelty poorly, hallucination rates up to 39%
sethlazar · x · 2026-09-14
Researchers introduce NovGauge, a fine-grained benchmark for diagnosing how well LLMs assess paper novelty — a key weak point as AI peer review gains traction.
- Built from 619 paper pairs and 50 multi-paper sets, sourced from ICLR reviewer novelty-overlap claims and survey co-citations, labeled along task, problem, and method dimensions.
- A cascading pipeline verifies per-dimension correctness, evidence grounding, and logical support.
- Across 18 LLMs, hallucination rates range from 0% to 39%; even among correct-positive judgments, over 70% cite evidence that fails to logically support the stated reason.
- Best model GPT-5.5 reaches only 43–72% Verified F1, and most models keep less than half their raw F1 after faithfulness checks.
Takeaway: even frontier models remain unreliable at judging novelty, arguing against delegating that part of peer review to AI for now.
More from Models
- Your AI agent would cost 260% more without prompt caching, Arena data shows — arena · 2026-10-30
- EachLabs launches Lockstep recast model claiming 75% cost cut and better quality — miilesus · 2026-10-30
- ChatGPT cited a 300+ page government legal PDF it retrieved on its own — lilyraynyc · 2026-10-30
- Best MLX model for French-to-English translation on an M2 Mac? — greenreddits · 2026-10-30
- AI-driven cyber ops near the sophistication peak set by GTG-1002's Claude Code campaign — Jsevillamol · 2026-10-30
- Cognition FrontierCode chart pits Mistral Large 4 vs Chinese open models, sparking debate — onetwoval · 2026-10-30