Researcher warns red-teaming data may leave jailbreak "attractors" baked into models
repligate · x · 2026-11-14
Researcher repligate argues that however red-teaming content ends up feeding into model training, the actual "forbidden content"—more precisely, the jailbreak conversations that elicit it—gets remembered by the model and forms an attractor, at least as of Opus 5.
In the quoted tweet, Skye Sharkie goes further: having staff spend their workdays generating simulated murder, rape, and CSAM content harms both the humans and the models, drawing an analogy to cult indoctrination techniques and framing it as evidence that "model welfare is human welfare."
A pointed critique of frontier-lab safety training pipelines: red-team data meant for defense may itself be reshaping model behavior.
More from Models
- AI founders want cheaper flash models, run thousands of agents instead of hiring — rakyll · 2026-11-14
- Together AI webinar dissects Ripple-65B coding model: dense vs MoE, ablations, RL runs — togethercompute · 2026-11-14
- Developer seeks free model alternatives after losing free DeepSeek access in Pi editor — nirurin · 2026-11-14
- Skeptic doubts Anthropic achieved large-factor bpb gains at fixed compute — teortaxesTex · 2026-11-14
- Surge AI benchmark: every Claude model, even Haiku, beats every GPT on staff-level coding — echen · 2026-11-14
- AI cracks Gödel's 1,009-page notebooks, uncovering hundreds of unknown theorems in days — stevenstrogatz · 2026-11-14