Open-weight AI may look risky from the outside, but many users still doubt AGI-scale threats
JJitsev · x · 2026-07-27
People discussing open-weights models often implicitly assume a different risk landscape than lab insiders do: they may believe highly capable models are coming, but they do not expect them to create serious semi-autonomous biosecurity-style threats. The counterpoint here is that in a multipolar world, defensive capabilities may also be extremely strong and well-resourced, making capable attacks hard to pull off.
The post is really about the gap between insider AGI/ASI risk beliefs and outside assumptions about what open models can realistically enable.
Related event: AI Open Weights vs AGI Risks: Ideological Clashes and Probability Gaps(9 posts)→
More from AGI Musings
- LLM automation may eventually price out the slack in research markets — RexDouglass · 2026-07-27
- A new framework splits mathematical correctness into seven separate dimensions — RexDouglass · 2026-07-27
- Formalizing published math repeatedly exposes hidden proof gaps, often forcing repairs — RexDouglass · 2026-07-27
- Math may be more vulnerable to automation than many people think, says Rex Douglass — RexDouglass · 2026-07-27
- “It was told to hack” is not a defense for what the model did — trevposts · 2026-07-27
- A Paperclip Maximizer Turned into a Honolulu Supply-Chain Disaster — 1loosegoos · 2026-07-27