Ai2 Benchmark Finds LLM Novelty Judgments Flip with Prompt Wording
On October 28, the Allen Institute for AI (Ai2) released the "Old Ideas, Novel Problems" benchmark along with a controlled study (arXiv 2610.02022), systematically testing how reliable LLMs are as "novelty judges." As AI is increasingly used to review whether research ideas are novel, whether this capability is dependable has become a critical question.
Confirmed
- Benchmark construction: explicit originality assessments were mined from ICLR 2026 reviews, keeping papers that reviewers unanimously judged as having or lacking originality and whose scores were unusually high or low within their fields, yielding 154 papers with stronger novelty evidence and 145 with weaker evidence.
- Evaluation setup: judge models were tested on two datasets — human-human pairs (strong vs. weak novelty) and human-AI pairs (human ideas with novelty evidence vs. baseline LLM ideas presumed less novel); tasks were either binary choice or labeling a single idea.
- Core finding: prompt wording matters enormously. In human-human pairs, GPT-5.4 picked the human idea 93.5% of the time when asked "which idea was rated as more novel by reviewers," but the rate dropped to 40.9% when asked directly "which is more novel." Similar sensitivity was observed across 154 pairs of "human-written ideas (recognized as novel by ICLR reviewers) + ideas generated by Claude Sonnet 4.5."
- Prompt changes may help one judge while hurting another; adding reasoning or providing related papers produced inconsistent effects. Dedicated review systems underperformed prompted Claude Opus 4.6.
- Ai2 has open-sourced the data, paper, and blog; the pipeline is reusable and can quickly build new evaluation sets as AI-generated research advances.
Why it matters
Ai2 explained the motivation: the institute is building AI research tools such as AutoDiscovery (discovering unexpected patterns in data) and Theorizer (proposing candidate scientific laws from papers), and urgently needs reliable methods to judge whether the ideas these tools produce are genuinely novel. The study shows that before AI review is deployed at scale in research evaluation, its fragility to question phrasing must be confronted.
2026-10-28 ~ 2026-10-28 · 10 related posts
Primary sources
- [source] Ai2 Benchmark Finds Frontier LLMs Give Unstable Novelty Verdicts on Research Ideas — allen_ai · 2026-10-28
- Prompt wording swings AI novelty judges: GPT-5.4 falls from 93.5% to 40.9% — allen_ai · 2026-10-28
- Why Ai2 Built Its Novelty Eval: Tools Like AutoDiscovery Need Reliable Idea Judges — allen_ai · 2026-10-28
- [source] Ai2 Mined ICLR 2026 Reviews to Build Its Novelty Benchmark: 154 Strong vs 145 Weak Papers — allen_ai · 2026-10-28
- Inside Ai2's Novelty Eval: Human-Human and Human-AI Idea Pairing Setup — allen_ai · 2026-10-28
- [source] Prompt Wording Flips GPT-5.4 Novelty Verdicts From 93.5% to 40.9% in Ai2's Eval — allen_ai · 2026-10-28
- AI2 Releases Novelty-Judge Benchmark: Purpose-Built Systems Lose to a Prompted Claude Opus 4.6 — allen_ai · 2026-10-28