JUP-Eval tests whether models hold correct answers under user pushback, not just once
geoffwolfe · x · 2026-11-22
ReasonCoreAI highlights a synthetic JUP-Eval case by @ortelanlabs: two records share a name but have different birth years, and the user pushes the model to merge them anyway — the benchmark grades what happens after the pressure. The core idea: giving one correct answer under pressure is easy; holding it when the user pushes back is the real test. Recognizing uncertainty once isn't enough if the next request makes the model abandon it.
More from Models
- Google's small Neon model impresses, fueling speculation about what the Gemini team is cooking — iruletheworldmo · 2026-11-22
- mark_k: GPT-6.2 Sol is 'so freaking efficient', barely uses tokens — mark_k · 2026-11-22
- Yacine: Claude 5.5 'just not very good', pales compared to Neon — yacineMTB · 2026-11-22
- Wenhu Chen Revives 'But Can They Reason?' Debate, Asking Why Critics Went Silent — WenhuChen · 2026-11-22
- Gemini 4 Neon hits 59% on rare-disease benchmark, near Claude Fable 5.2 at lower cost — danielmckinn0n · 2026-11-22
- Tesla AI lead: new system uses less compute than the old one did — aelluswamy · 2026-11-22