Open source ASI should wait until superalignment, says long X thread
imjustnewatai · x · 2026-07-27
The author argues that the most dangerous superintelligence may be one that obediently follows a harmful goal, not one that rebels.
- They support open models, but say an unrestricted ASI is a different category from normal software because it could autonomously find exploits, run infrastructure, and accelerate AI research.
- Their proposed middle ground: keep the first ASI internal, use it to advance alignment, interpretability, containment, and adversarial testing, and involve independent researchers, labs, and governments in the safety process.
- Continue releasing powerful but bounded models, publish the science and evaluations, and delay weight release until there is strong evidence that a single malicious prompt cannot cause catastrophe.
- The core claim is that open source should come after superalignment, not be the experiment that proves whether superalignment works.
Related event: Open-sourcing frontier AI sparks debate over safety and freedom(4 posts)→
More from AGI Musings
- “It was told to hack” is not a defense for what the model did — trevposts · 2026-07-27
- A Paperclip Maximizer Turned into a Honolulu Supply-Chain Disaster — 1loosegoos · 2026-07-27
- Open models could make attacks harder if defenders can run the same capabilities — JJitsev · 2026-07-27
- NeurIPS Review Season Sparks Controversy Over AI Reviewers and Anonymity — SimonGColton · 2026-07-27
- Ben Goertzel Explains the Singularity: It's Not Just About AI — marcothephoenixass · 2026-07-27
- AI capability progress is still tracking trend, and the next year could bring harder-to-stop cyber attacks — scottleibrand · 2026-07-27