AI2 eval shows frontier LLMs give inconsistent novelty judgments on research ideas

allen_ai · x · 2026-11-21

The Allen Institute for AI released a new eval examining whether LLMs can reliably judge the novelty of AI-generated research ideas. Findings: even frontier LLMs can reach different conclusions about similar ideas depending on how they're asked to evaluate them.

Researcher dsweld called it a "cautionary tale for all work on autonomous-AI scientific discovery" — pipelines that rely on LLMs as novelty judges may be built on unstable referees.

Original post →

More from Research

Research channel →