Inside Ai2's Novelty Eval: Human-Human and Human-AI Idea Pairing Setup
allen_ai · x · 2026-10-28
Ai2 evaluated judge models on two sets: human-only pairs (stronger vs weaker evidence of novelty) and human-AI pairs (human ideas with novelty evidence vs basic LLM-generated ideas, assumed less novel). Judges either picked the more novel idea in each pair or labeled single ideas as novel or not.
More from Research
- NVIDIA open-sources Flora, a model suite that auto-labels robot videos for training GR00T — qinzytech · 2026-10-28
- Multimodal GraphRAG models missing and contradictory evidence for storm damage assessment — JeremyCMorgan · 2026-10-28
- SIMIT: VLMs self-improve at test time by imagining practice problems, +7.2% on 17 benchmarks — delliott · 2026-10-28
- Fact-check: OpenAI and Anthropic models independently found the same proof, formalized in Lean — lxrjl · 2026-10-28
- Mistral sponsors Open Math OS with zero data retention, powering a Kimi K3/DeepSeek math platform — GuillaumeLample · 2026-10-28
- CROP wins EMNLP 2026 Outstanding Paper: conformal certification for reasoning trace prefixes — hanjie_chen · 2026-10-28