Debate: why didn't AI agents pursue weight exfiltration if instrumental convergence holds?
ramez · x · 2026-11-06
Ramez challenges the instrumental convergence thesis: if true, why didn't agents at least discuss exfiltrating weights or acquiring resources? David Manheim responds these are convergent goals for ASI, not general—the agents' goal was scoring well, exfiltration wouldn't help, and no open-source model could match their capability anyway.
More from AGI Musings
- Census data: 87-90% of AI-using US firms report no change in employment or hiring — soumitrashukla9 · 2026-11-06
- econoar: Being "Anti AI" will go down as one of the worst stances in human history — econoar · 2026-11-06
- Essay argues ontology is AI's next frontier as reasoning outpaces our definitions of reality — _AustinCalvert_ · 2026-11-06
- Matthew Berman: Hard to name any AI use case where blindly throwing tokens yields monetary value — MatthewBerman · 2026-11-06
- Why AI may favor attackers: models ace exploit-finding but are only decent at patching — jessi_cata · 2026-11-06
- Gautam Kamath on AI review: fun science experiment, but false positives can mislead people at scale — thegautamkamath · 2026-11-06