JUP-Eval tests whether models hold correct answers under user pushback, not just once

geoffwolfe · x · 2026-11-22

ReasonCoreAI highlights a synthetic JUP-Eval case by @ortelanlabs: two records share a name but have different birth years, and the user pushes the model to merge them anyway — the benchmark grades what happens after the pressure. The core idea: giving one correct answer under pressure is easy; holding it when the user pushes back is the real test. Recognizing uncertainty once isn't enough if the next request makes the model abandon it.

Original post →

More from Models

Models channel →