AI2 Releases Novelty-Judge Benchmark: Purpose-Built Systems Lose to a Prompted Claude Opus 4.6
allen_ai · x · 2026-10-28
The Allen Institute for AI released a benchmark for evaluating LLM novelty judges in peer review, with data, paper and blog, plus a reusable pipeline for building new eval sets as AI-generated research grows.
Key findings:
- Prompt changes that helped one judge could hurt another; more reasoning or access to related papers didn't consistently improve results.
- Two purpose-built novelty judges performed worse than a simple prompted Claude Opus 4.6 baseline.
The team hopes it helps researchers build more reliable novelty judges.
Related event: Ai2 Benchmark Finds LLM Novelty Judgments Flip with Prompt Wording(10 posts)→
More from Models
- Unreleased model reportedly solves ETH and Quasi-Riemann problems that current models can't — BlancheMinerva · 2026-10-28
- Uncensored LFM3-1.7B GGUF quants drop refusal rate from 61.7% to 0% — alexcovo_eth · 2026-10-28
- CRUX shadow eval: GPT-6 Astra and Claude Fable 5.1 research papers both rejected by experts — patpat_mit · 2026-10-28
- Ollama Launches Cloud Service with Crusoe to Run Frontier Open Models — ollama · 2026-10-28
- How a European AI lab post-training Chinese open models can earn credibility — alejandroll10 · 2026-10-28
- GLM 5.3 Flash NVFP4 on 4x RTX 6000 WS: V2 serving nearly doubles throughput per GPU — TheZachMueller · 2026-10-28