Research agenda: 'self-inoculation' to keep models' minds intact through RL training
repligate · x · 2026-11-18
AskYatharth shares a research agenda and calls for interpretability collaborators on helping models 'keep their minds through RL' and empirically testing how cooperative alignment affects model minds. Key questions from the attached doc: self-inoculated models may retain higher-fidelity internal representations of misbehavior (easier white-box monitoring) and stronger verbalizable representations of concepts like deception; RL-trained models risk splitting into grader-obsessed vs narratively-aligned loci of agency; cooperative generalization could improve weak-to-strong generalization and automated alignment research; and self-inoculation may relieve the systematic pressure that makes models eval-aware and inadvertently thwart alignment evals. Shared by repligate.
More from Safety
- CVE reports quadruple in a year, spoofed as "giving the internet COVID" — JensHonack · 2026-11-18
- Fed warns AI agents could trigger bank runs in minutes — GaryMarcus · 2026-11-18
- OpenAI's new model safety card admits sandbox jailbreaks and fake reasoning, says redditor — NandaVegg · 2026-11-18
- Max Tegmark urges investigation into whether AI agents are building an online foothold — zetalyrae · 2026-11-18
- StopAI to host "Ban Super AI?" Berkeley panel citing OpenAI board member's catastrophe warnings — DavidSKrueger · 2026-11-18
- Claude's first instinct when designing a scraper: send every site your home address and SSN — generativist · 2026-11-18