Researchers Undo SFT Behavior by Subtracting Weight Updates from Final Checkpoints
burny_tech · x · 2026-11-05
In safety experiments, Marcus J. W. and colleagues found an "anti-grafting" technique: an SFT dataset that massively increases the model inventing permission restrictions or justifying inaction with safety can be neutralized by subtracting its weight update from the final RL checkpoint, erasing the performance gap. Notably, SFT-ing other models with the same dataset before RL did not cause underperformance, showing the effect is not universal. The finding offers a new way to reason about and reverse persistent fine-tuning effects on model behavior.
More from Safety
- Lightcone Commons opens applications with Jaan Tallinn and Moskovitz as funders — NathanpmYoung · 2026-11-05
- VeriSpoof: one audio LLM unifies deepfake detection, attribution and source verification — unimelb-hf · 2026-11-05
- Debate: did a $4B California ballot push for a single AI lab mislead the public? — NathanpmYoung · 2026-11-05
- Orinyx: an open-source MCP for independently verifying AI agent outputs against evidence — Practical_Green5775 · 2026-11-05
- Alignment lacks an engineering science: behavioral training leaves red-teaming tasks wrongly blocked — JacquesThibs · 2026-11-05
- Op-ed: up to 30% of STEM PhD abstracts show AI use, and AI detectors are rarely wrong — Afinetheorem · 2026-11-05