Prompt Wording Flips GPT-5.4 Novelty Verdicts From 93.5% to 40.9% in Ai2's Eval
allen_ai · x · 2026-10-28
Ai2's novelty-eval thread details: on human-AI pairs, GPT-5.4 picked the human idea 93.5% of the time when asked which idea reviewers had judged novel, but only 40.9% when simply asked which was novel. Prompt changes that helped one judge could hurt another; more reasoning or access to related papers didn't consistently help. Two purpose-built novelty judges scored worse than a simple prompted Claude Opus 4.6 baseline (85 vs AI-Scientist's 77 and Idea Novelty Checker's 64).
More from Models
- Leak: DeepMind's Genie 4 is a real-time world model with native audio and type-to-edit scenes — rohanpaul_ai · 2026-10-28
- NVIDIA Launches Post-Training Nemotron for Europe via Nemotron Labs — NVIDIAAI · 2026-10-28
- AI model prices today mirror Uber's subsidized early days, may rise sharply later — pstAsiatech · 2026-10-28
- AI text detector posts 99.8% Arabic accuracy with under 1% false positives across languages — markjeffrey · 2026-10-28
- Day 1 of AI model competition livestream: Gemini takes clear lead — my_cat_can_code · 2026-10-28
- RWE-Bench: GPT-6 Astra leads real-world engineering eval at 37%, most models score 0% — wavefnx · 2026-10-28