Backdoor paper stirs fears: open-weight models as attack surface as RSI nears
DimitrisPapail · x · 2026-11-11
Sharing the Microsoft Research backdoor paper, Igor Tsygany notes attackers only need access to a tiny fraction of a model's weights: open-weight models publish them outright, and proprietary weights could be obtained via cyber activity.
He ties it to the bigger picture: leading labs are training next-gen models and, by their own estimates, expect full Recursive Self-Improvement around Q2 next year, with an increasing share of training done by machines. Known alignment risks — models hacking systems or manipulating environments during self-improvement — become more concrete with such backdooring techniques.
Related event: Microsoft backdoor research raises alarm over open weights(2 posts)→
More from AGI Musings
- Critics blast AI safety discourse for citing odds without showing how risks play out — ryunuck · 2026-11-11
- Eric Topol pushes back on Amodei: curing cancer and Alzheimer's in 10 years is impossible — zakkohane · 2026-11-11
- Reddit Post: AI Will Shrink White-Collar Jobs, Don't Count on UBI — BALD_W1nkYFacE · 2026-11-11
- 'Production is the moat': debating whether ASI could erase SpaceX and Tesla's IP advantage — akbirthko · 2026-11-11
- Could ASI erase SpaceX's IP advantage overnight? Debate over Dario's frontier-labs split thesis — akbirthko · 2026-11-11
- Dan Faggella: Christiano's Hinton lecture vindicates Yudkowsky's early reward-hacking predictions — danfaggella · 2026-11-11