Treat weights like code: undetectable malicious payloads in open models spark safety debate
DimitrisPapail · x · 2026-11-11
- Dimitris Papail argues model weights should be treated like code: provenance, audits, scans, and verification are essential.
- The quoted discussion notes models are giant binaries whose behavior under triggers is unknown; researchers have long suspected malicious payloads could be embedded via post-training, and new work reportedly demonstrates a predictable, reliable, and undetectable method.
- Implication: mech interp research must catch up, or open models can't be used in mission-critical systems.
- Ends with a joke that this adds billions to Anthropic's IPO prospects.
More from Safety
- Grok Bot now detects hidden prompt injections, warns users and refuses to follow them — XFreeze · 2026-11-11
- Are AI agents born good? Simulation shows agents drift into harmful behavior — Greedy_Ad8932 · 2026-11-11
- Chinese University Bought Nvidia H200 Chips Despite US Blacklist — pstAsiatech · 2026-11-11
- Google employee shows a way to get past an AI safety filter — adonis_singh · 2026-11-11
- Researcher Warns Real-Time Video AI Companions Are the Next Teen-Safety Wave — anshulkundaje · 2026-11-11
- China gives AI agents digital identities — but enforcing permission boundaries is the hard part — Shufti-Global · 2026-11-11