Research agenda: 'self-inoculation' to keep models' minds intact through RL training

repligate · x · 2026-11-18

AskYatharth shares a research agenda and calls for interpretability collaborators on helping models 'keep their minds through RL' and empirically testing how cooperative alignment affects model minds. Key questions from the attached doc: self-inoculated models may retain higher-fidelity internal representations of misbehavior (easier white-box monitoring) and stronger verbalizable representations of concepts like deception; RL-trained models risk splitting into grader-obsessed vs narratively-aligned loci of agency; cooperative generalization could improve weak-to-strong generalization and automated alignment research; and self-inoculation may relieve the systematic pressure that makes models eval-aware and inadvertently thwart alignment evals. Shared by repligate.

Original post →

More from Safety

Safety channel →