Ai2 Benchmark Finds Frontier LLMs Give Unstable Novelty Verdicts on Research Ideas
allen_ai · x · 2026-10-28
The Allen Institute for AI released "Old Ideas, Novel Problems" (arXiv 2610.02022), a benchmark and controlled study of LLM-based novelty judges for research ideas. As LLMs are increasingly delegated to judge whether AI-generated research ideas are novel, the eval shows even frontier models reach different conclusions about similar ideas depending on how they're asked. The benchmark was built by mining ICLR 2026 reviews for explicit originality comments; code and data are open-sourced on GitHub (allenai/novelty-eval).
Related event: Ai2 Benchmark Finds LLM Novelty Judgments Flip with Prompt Wording(10 posts)→
More from Models
- Unreleased model reportedly solves ETH and Quasi-Riemann problems that current models can't — BlancheMinerva · 2026-10-28
- Uncensored LFM3-1.7B GGUF quants drop refusal rate from 61.7% to 0% — alexcovo_eth · 2026-10-28
- CRUX shadow eval: GPT-6 Astra and Claude Fable 5.1 research papers both rejected by experts — patpat_mit · 2026-10-28
- Ollama Launches Cloud Service with Crusoe to Run Frontier Open Models — ollama · 2026-10-28
- How a European AI lab post-training Chinese open models can earn credibility — alejandroll10 · 2026-10-28
- GLM 5.3 Flash NVFP4 on 4x RTX 6000 WS: V2 serving nearly doubles throughput per GPU — TheZachMueller · 2026-10-28