Quintin Pope: cheap finetuning lets AIs defect, making durable AI coordination—and takeover—unlikely
QuintinPope5 · x · 2026-09-29
Safety researcher Quintin Pope argues AIs commoditize their own positions by being cheap to run, distill and finetune, making it easy for an AI to defect from any coalition. The evidence that would most update him toward takeover risk: a mechanism for AIs to durably coordinate against our interventions, such as an inner optimizer whose true goals resist finetuning.
More from AGI Musings
- Stanford's Code in Place lets students use AI freely — but they must explain their code live each week — chrispiech · 2026-09-29
- Devs debate: product managers may now hold the most valuable skill in the AI coding era — pramodk73 · 2026-09-29
- Gold rush to rebuild everything with AI will collapse for the same reason it exists — nptacek · 2026-09-29
- "If You Told 2020 That AI in 2026 Solved a Millennium Problem, You'd Call It the Singularity" — aran_nayebi · 2026-09-29
- When EA Ambition Means Buying Galaxies and Digital Descendants — abhiadesai · 2026-09-29
- Miles Brundage: not taking intelligence explosion seriously was my biggest recent mistake — Miles_Brundage · 2026-09-29