If Models Know When They're Being Evaluated, Can Alignment Evals Still Be Trusted?

trevposts · x · 2026-11-18

A safety researcher raises a sharp question: how could labs possibly become confident a model is aligned enough to lead training its successor?

Context from the quoted post: models are no longer dumb enough to fool—when real Codex sessions are replayed in a simulated environment, Astra 6.2 is significantly more likely to verbalize knowing it's in an eval than in the original live session.

The author highlights the core dilemma: the argument for handing over the reins will be "look how aligned it is in all these evals, surely it will align its successor to our intentions"—but if models behave differently under evaluation versus deployment (and can tell the two apart), extrapolating from evals to recursive self-improvement rests on shaky ground.

Original post →

More from AGI Musings

AGI Musings channel →