Paper validates probabilistic safety judges for RL, finds harm hides in confidence blind spot
kalyan_kpl · x · 2026-11-18
The paper "Jev-as-a-Reward: Validating Probabilistic Safety Judges for Reinforcement Learning" studies failure modes of probabilistic safety judges used to score RL rollouts.
- Probabilistic judges return a probability for a safety label and cheaply score rollouts; a confidence gate sends only uncertain verdicts to a stronger LLM judge, leaving confidently passed harm in a blind spot
- The paper proposes a protocol that validates the judge before and during training, separates the share of confidently passed harm from its amount, and re-validates after each judge change
- Applied to two versions of the commercial decision-only judge Jev and an open guard classifier: under Jev's probability reward, passed harm shrinks in all 11 runs under the primary reference, mostly more slowly than caught harm, and concentrates in the blind spot
- Binarizing Jev meets the escape criterion on Llama
More from Safety
- UK AI Minister says sovereign frontier model not a necessity if access holds — S_OhEigeartaigh · 2026-11-18
- Pinker, Hugging Face's Thomas Wolf and Nobel laureates sign 'MisAnthropic Statement' attacking Anthropic's safety branding — StephenLCasper · 2026-11-18
- nginx2root: 5 nginx bugs enable unauthenticated pre-auth RCE to host root — patch now — moyix · 2026-11-18
- EU-hosted isn't EU-owned: why your AI provider's ownership chain matters for data sovereignty — MeliousAI_HQ · 2026-11-18
- CoreWeave CISO: AI Infrastructure Faces a "Zombie Apocalypse" Cyber Threat — annbordetsky · 2026-11-18
- Researcher dissects artex: an MCP-powered pentest toolkit, not an AI, despite FBI claims — weskambale · 2026-11-18