141,750 Explanations Tested: LLM Denial Reasons Are True but Fixes Still Fail

alex_verem · x · 2026-11-18

The arXiv paper "Right Reason, Wrong Advice" systematically evaluates LLM-written denial explanations, introducing recourse validity: when multiple rules independently deny a request, citing one true reason leaves the applicant denied after fixing it.

Setup: 20 LLMs tested across 7 rule systems ported from real ones (SNAP, public housing, federal student aid, access control), yielding 141,750 explanations.

Key findings

Context: US states are testing AI-drafted food-aid denial notices; the author argues such notices should be judged solely by whether acting on them gets people approved.

Related event: Study: LLM Denial Explanations Are Right but Advice Fails(2 posts)→

Original post →

More from Safety

Safety channel →