repligate: A superhuman-coding AI was the classic X-risk scenario — now it's here
repligate · x · 2026-09-26
repligate observes that under classic Yudkowskian standards, an AI superhuman at coding in most domains, autonomous for days, and able to find 0-days in major systems would have been judged a direct existential risk — and such systems are now nearly here. He adds that people at Anthropic reportedly take the threat model of current or near-future models killing or disempowering humanity seriously, which he finds reasonable. The post highlights the mismatch between safety-community risk thresholds and how little societal response has changed as capabilities cross them.
Related event: repligate: Superhuman Coding AI Is Here, Yet No Existential Risk(2 posts)→
More from AGI Musings
- Embedded AI lab evaluators beat nothing, but audits need government teeth: Atlantic essay — ghadfield · 2026-09-26
- A $500k engineer costs $2k/day — $200 of LLM tokens buying 20% output is easy ROI — generativist · 2026-09-26
- Katja Grace: If you want an AI utopia, don't pursue it via a high-risk reckless route — KatjaGrace · 2026-09-26
- Paul Graham's essay on involuntary thinking resurfaces as AI amplifies idea exploration — aminkarbasi · 2026-09-26
- Google engineer Robert O'Callahan quits AI chip team, warning AI is progressing too fast — Polymarket · 2026-09-26
- lateinteraction: with 1B agents, at least one hacking something is statistically inevitable — lateinteraction · 2026-09-26