Ai2 eval: frontier LLMs give inconsistent verdicts on research idea novelty
allen_ai · x · 2026-10-28
Allen Institute for AI released a new eval probing whether LLMs can reliably judge the novelty of AI-generated research ideas — an increasingly common role in AI-assisted research. Findings: even frontier models reach different conclusions about similar ideas depending on how they're asked to evaluate, raising concerns about their reliability as novelty judges.
Related event: Ai2 Benchmark Finds LLM Novelty Judgments Flip with Prompt Wording(10 posts)→
More from Models
- Unreleased model reportedly solves ETH and Quasi-Riemann problems that current models can't — BlancheMinerva · 2026-10-28
- Uncensored LFM3-1.7B GGUF quants drop refusal rate from 61.7% to 0% — alexcovo_eth · 2026-10-28
- CRUX shadow eval: GPT-6 Astra and Claude Fable 5.1 research papers both rejected by experts — patpat_mit · 2026-10-28
- Ollama Launches Cloud Service with Crusoe to Run Frontier Open Models — ollama · 2026-10-28
- How a European AI lab post-training Chinese open models can earn credibility — alejandroll10 · 2026-10-28
- GLM 5.3 Flash NVFP4 on 4x RTX 6000 WS: V2 serving nearly doubles throughput per GPU — TheZachMueller · 2026-10-28