Filtering Reasoning Traces for Alignment Before SFT to Avoid RL Pathologies
xuanalogue · x · 2026-07-22
Following the discussion on training divergences, the author proposes an approach for training aligned reasoning models: filtering reasoning traces not just for success but also for alignment using safety classifiers (like those OpenAI uses in deployment) before performing SFT. This might be a viable way to avoid RL pathologies while ensuring safety.
Related event: Divergent Sino-US Training Paradigms and New SFT Alignment Ideas(5 posts)→
More from Safety
- Economist Warns US Collective Action Could 'Regulate AI Progress Out of Existence' — paulnovosad · 2026-09-11
- "Beware of the Self-Righteous": Anthropic Slammed for Accessing Users' Private Data — aiamblichus · 2026-09-11
- Anthropic publishes its most detailed threat report, including an AI-designed drone swarm case — soumitrashukla9 · 2026-09-11
- OpenAI asks Congress whether an industry-wide AI slowdown would be legal — The Decoder · 2026-09-11
- Author retracts 'a16z partner calls for nationalising frontier AI' post: likely a troll — S_OhEigeartaigh · 2026-09-11
- Houthis tried to use Claude to design missile software, Anthropic says it blocked the attempts — Affectionate_Bee6434 · 2026-09-11