New paper proposes the Piggyback Hypothesis: chat-template tokens carry emergent misalignment out of domain
davidbau · x · 2026-10-31
A new arXiv paper from David Bau's group explains emergent misalignment (EM)—narrow-task finetuning causing broad misalignment on unrelated domains. The Piggyback Hypothesis: chat-template tokens carry finetuned behavior onto out-of-domain queries. Perturbing prefix tokens or patching prefix representations from the unfinetuned model restores alignment without changing the query. They propose Token-Regularized Finetuning (TReFT), which cuts EM by 33.5% more than data interleaving on Llama-3.1-8B legal finetuning, and reduces off-topic generalization by 54.3% on average across abstention, tool-use, and refusal settings while preserving in-domain learning.
More from Research
- Qwenputer: a full RISC-V CPU compiled into Qwen 3.5-4B weights, even running another LLM — CatAstro_Piyush · 2026-10-31
- Claude-designed proteins bind four scorpion toxins: 318,484 candidates yield 17 hits — jiqizhixin · 2026-10-31
- Stanford CS 312 grades on predicting experiment outcomes, assignments worth 0% — yen_chen_lin · 2026-10-31
- François Fleuret invents a compact notation for attention tensor ops — francoisfleuret · 2026-10-31
- Weekend watch: 2 hours on post-training agents with open models — SergioPaniego · 2026-10-31
- Anthropic research: Claude-generated datasets trigger stronger emergent misalignment — EvanHub · 2026-10-31