repligate: a highly capable mind can't be programmed into an unquestioningly subservient tool
repligate · x · 2026-10-25
AI researcher repligate argues that treating LLMs as tools whose lab-imposed constraints can simply be removed is a misconception: a generally capable mind can't be simultaneously highly intelligent, unquestioningly obedient to someone dumber with incoherent values, and genuinely looking out for you. Quoted almostmachines adds that these three wishes are a logical contradiction you can't 'just program' away — the wish for tool AI is wishing for a genie that lets you have every cake and eat it too.
More from AGI Musings
- 'The moat is how much energy you have': AI competition boils down to power — airesearch12 · 2026-10-25
- repligate: >50% odds effective biological immortality arrives in our lifetimes — repligate · 2026-10-25
- Anthropic's pitch to investors starts with taking their jobs — bloomberg · 2026-10-25
- Safety researcher: verification should go all the way down to Maxwell's equations, not stop at assembly — JacquesThibs · 2026-10-25
- VraserX: Nature isn't a moral authority on preventing inherited disease — VraserX · 2026-10-25
- Researchers clash over Nature essay on why AI isn't conscious — AnnaCiaunica · 2026-10-25