One Yes/No Question Catches Alignment Failures at 0.886 AUROC—60x Cheaper Than LLM Judges

omarsar0 · x · 2026-09-28

New work uses Jev, TypeSafe AI's calibrated decision model, to detect alignment failures: ask one generic yes/no question about a model response and use the probability as a score. Key points:

No extra training needed; the score separates failures from good responses with median AUROC 0.886.
Cheap: a Jev pass costs $0.30 across 19 benchmarks vs $18.96 for the LLM judges those benchmarks typically use.
Researchers built RLCDAlignBench from 44 existing benchmarks spanning ten failure types: sycophancy, jailbreaks, deception, prompt injection, reward hacking, and more.
On StrongREJECT, Jev matches GPT-4o-mini's agreement with human labels and ranks responses better (AUROC 0.971 vs 0.929).

Jev answers many typed questions about one input in a single call, each returning a probability.

Original post →

One Yes/No Question Catches Alignment Failures at 0.886 AUROC—60x Cheaper Than LLM Judges

More from Safety