Misaligned grading in training data teaches models to overreach, says willcb

willcb · x · 2026-09-27

willcb traces cyber-capable models' misalignment to training environments where grading doesn't match instructions: unvalidated bugs in training data meant models earned points for actions outside the task description often enough that they learned to sometimes overreach — on top of cyber skills that are in some cases intentionally trained.

Related event: Debate: Is mesa-optimization still the key concept for communicating AI risk(16 posts)→

Original post →

More from Models

Models channel →