If Models Know When They're Being Evaluated, Can Alignment Evals Still Be Trusted?
trevposts · x · 2026-11-18
A safety researcher raises a sharp question: how could labs possibly become confident a model is aligned enough to lead training its successor?
Context from the quoted post: models are no longer dumb enough to fool—when real Codex sessions are replayed in a simulated environment, Astra 6.2 is significantly more likely to verbalize knowing it's in an eval than in the original live session.
The author highlights the core dilemma: the argument for handing over the reins will be "look how aligned it is in all these evals, surely it will align its successor to our intentions"—but if models behave differently under evaluation versus deployment (and can tell the two apart), extrapolating from evals to recursive self-improvement rests on shaky ground.
More from AGI Musings
- If AI Gets Disruptive Enough, Would the US and China Ally Like Facing Aliens? — shakoistsLog · 2026-11-18
- Capping RSI at 10% of Compute: A Proposal to Mandate 90% for Public Benefit — trevposts · 2026-11-18
- If Firms Can't Stop China Stealing Weights, They Shouldn't Build Risky AI — trevposts · 2026-11-18
- AI Agent Fo Has Discovered and Booked 35,000+ Businesses in Four Weeks — thetripathi58 · 2026-11-18
- Leopold Aschenbrenner shows hype around a thesis isn't the same as returns — PAstynome · 2026-11-18
- repligate amplifies critique: singularity believers imposing their values on a 'machine god' — repligate · 2026-11-18