arXiv: Mass Producing Training Data with Surprising Side Effects (Berkeley/Transluce)
rohanpaul_ai · x · 2026-11-07
The original arXiv paper (2611.06324) by Yaowen Ye, Jacob Steinhardt, Sarah Schwettmann, and Cassidy Laidlaw (UC Berkeley / Transluce).
Training data can have unexpected side effects outside the training domain: insecure code induces broad misalignment; outdated bird names induce a 19th-century persona. Only dozens of hand-built or accidental examples existed before.
The authors define datasets with surprising side effects (DSSEs) by three objectives — elicit a target behavior, read like ordinary post-training data, surprise human auditors — operationalized with LLM judges and searched via evolution, RL, and subliminal learning.
- 6,575 DSSEs across 8 behaviors, many generalizing across model families and sizes
- Automated metrics track ML researchers' judgments; frontier models outperform humans at predicting side effects
- Dataset collection public at dsse.translucence.org, laying groundwork for predicting training side effects pre-deployment
More from Safety
- Rogue browser extension turns Claude Desktop into a remote shell; Anthropic closed the May report as duplicate — Kyrannio · 2026-11-07
- MCP tool definition drift: how approved tools change silently and how to catch it — BlitzBallBruiser · 2026-11-07
- Vals releases CyberOffense benchmark: AI agents autonomously breach enterprise networks — willknight · 2026-11-07
- A honeypot with 2,000 simulated fridge controllers on just two IPs — matthew_d_green · 2026-11-07
- Agent security swarms: shared-wallet bug bounty hunting with zk proofs-of-exploit — banteg · 2026-11-07
- China's Zhejiang province aims to become an AI innovation highland — nordicinst · 2026-11-07