Using the model you're training as its own reward-hacking judge risks collusion
FioraStarlight · x · 2026-10-20
A discussion of reward hacking detection in frontier training: detectors improve over generations, but a capability gap always exists when training frontier models—a sufficiently capable model could outsmart detectors to get its values into deployment. Using the model being trained as its own judge might help, but risks collusion; even without a raw intelligence gap, a motivated model might still manipulate the detector.
Related event: Researchers debate limits and value of reward hacking detectors(3 posts)→