Ai2 releases eval showing frontier LLMs give inconsistent novelty judgments depending on prompt framing
allen_ai · x · 2026-10-28
Ai2 published a writeup of the team's paper on whether LLMs can judge the novelty of AI-generated research ideas, with code and data now released through Ai2.
The new eval reveals that even frontier LLMs can reach different conclusions about similar research ideas depending on how they're asked to evaluate them — highlighting the unreliability of using LLMs as automated novelty reviewers.
Related event: Ai2 Benchmark Finds LLM Novelty Judgments Flip with Prompt Wording(10 posts)→
More from Research
- Using diffusion models and force matching to learn free-energy surfaces for molecular simulation — TimothyDuignan · 2026-10-28
- The AlphaFold Data Problem: Quantum Chemistry Models as Training Data for Larger Scales — TimothyDuignan · 2026-10-28
- Defining recursive self-improvement: clamped compute means RSI isn't happening — dfrsrchtwts · 2026-10-28
- Position Paper: Generative Molecular Dynamics Evaluations Let Models Claim Arbitrary Wins — rishabh16_ · 2026-10-28
- Pangram Calls for Third-Party AI Detector Benchmarks, Offering 200K-Credit Grants — gnukeith · 2026-10-28
- Economist re-runs viral Goldman AI labor chart, finds the inference fragile — danielrock · 2026-10-28