AI safety debate: dangerous AI must self-bootstrap physical affordances, and capability scaling is outpacing skeptics
lu_sichu · x · 2026-09-14
lusichu pushes back on "AI can't do this now, so don't worry": startups are already trying to give AI physical-world capabilities, and a truly dangerous AI would be able to synthesize and bootstrap its own physical affordances. Given how fast capabilities are scaling (math as an example), waiting until AI can do something means never having time to prevent it.
In the quoted context, JoshPurtell counters: imagine training without code data—true cyber catastrophes likely wouldn't happen since misaligned AI would need many generalization-heavy steps to succeed. He adds that math is heavily trained on, that in-distribution vs out-of-distribution tasks diverge sharply as RL compute grows, and questions whether AI could build fabs cheaper or better than Musk or China.
More from AGI Musings
- Jensen Huang: next 2 decades of progress may exceed all of history combined — rohanpaul_ai · 2026-09-14
- GPT-4o psychosis snippets eerily resemble SCP wiki stories, likely in training data — code_star · 2026-09-14
- Vals AI: frontier labs shouldn't grade their own frontier; models may match researchers by Aug 2027 — JenniferHli · 2026-09-14
- Running Out of AI: How Usage Limits Give Ideas Time to Develop — danshipper · 2026-09-14
- Eichengreen: debt markets, not stocks, will drive the aftermath of the AI bubble — TobyWalsh · 2026-09-14
- Vitalik Buterin: Adversarial mechanism design could be AI safety's killer app — allisondman · 2026-09-14