Interpretability researchers debate whether early monosemantic features persist through RLHF
davidmanheim · x · 2026-10-20
David Manheim sparked a discussion on X: regarding Nick Cammarata's recent claims about (disputedly monosemantic) features that emerge early and evolve later in training, a key open question is whether the claimed monosemanticity persists through RLHF/RL post-training or shifts. He asked Neel Nanda and other interpretability researchers for thoughts on implications for Anthropic's new reward hacking avoidance method, and whether that approach could be more reckless than current ways of reducing reward hacking.
More from Safety
- Vals AI: DeepSeek V4.1 Flash Beats GPT-6 Sol on Offensive Cybersecurity Benchmark — Hesamation · 2026-10-20
- Microsoft expands Sovereign Cloud capabilities in Paris for sensitive workloads — satyanadella · 2026-10-20
- MARS steers multimodal safety using only text-derived refusal directions, no multimodal safety data needed — MHUGLab · 2026-10-20
- Lenovo paper: safety benchmark scores mostly measure labeling policy, not detection capability — Vahid Jebraeeli · 2026-10-20
- Shipping AI into 5 major banks without the 12-month compliance wait — Cisco's Al Chen — AI Engineer · 2026-10-20
- Microsoft 365 Copilot leaked data from unopened emails, reviving the 1988 confused deputy problem — YvesMulkers · 2026-10-20