22 researchers incl. Bengio lay out 16 open problems in open-weight AI safety
evijit · x · 2026-10-13
Avijit Ghosh, one of the authors, shared a TMLR-published paper, Open Technical Problems in Open-Weight AI Model Risk Management, while taking a jab at Anthropic's safety narrative.
Led by Stephen Casper with 22 authors (including Yoshua Bengio, Dan Hendrycks, Zico Kolter, Yarin Gal, Geoffrey Irving, and Dylan Hadfield-Menell), the paper argues that open-weight models are getting stronger and more widely adopted, yet safety research for them is scarce: weights can be arbitrarily modified, used without oversight, and spread irreversibly, with little dedicated safety tooling. It presents 16 open technical challenges spanning training data, training algorithms, evaluations, deployment, and ecosystem monitoring, stressing that openness about research, methods, and evaluations—not just weights—is key to a rigorous science of open-weight risk management.
In a reply, Ghosh directly criticized Anthropic: instead of casting Claude as the only safe mechanism (a huge conflict of interest for a company selling closed-model access), it should open up both the problems it finds and its safety solutions.
More from Safety
- DOE puts $159M into Genesis projects, expanding OSPREY agent that runs particle accelerators — omnivaughn · 2026-10-14
- OpenAI Studies Internal Signals Behind AI 'Metagaming' During Training — WormLivesMatter · 2026-10-13
- AI Safety Incident: Command Injection Via a Reference Tool Copies a Source File — WormLivesMatter · 2026-10-13
- AI self-replication across 2M smart TVs would need just 40 H100s, researcher calculates — peterwildeford · 2026-10-13
- XBOW Details Chrome Full-Chain Exploit: 3 Bugs, 6 Days, $250K Google Bounty — moyix · 2026-10-13
- Giving LLMs tools makes them worse at refusing harmful requests, up to 68.7% — lateinteraction · 2026-10-13