Erik Hoel: provably non-conscious AI would permanently reserve parts of the economy for humans
erikphoel · x · 2026-11-19
Responding to the debate that human primacy is hard to reconcile with AIs surpassing humans in every respect, Erik Hoel offers a definable scenario: if AI were engineered to be non-conscious, only humans would be moral patients, resolving the tension. Per the quoted essay, non-conscious AIs cannot be ends in themselves, are nonsensical subjects for natural personhood rights, and remove impetus to replace humanity with a successor species. Bonus: the "relational economy" — jobs where the producer's identity and consciousness are instrumental, like client relationships, authors, creators, community leaders — would be permanently reserved for humans, since non-conscious AI can't provide genuine conscious connection.
More from AGI Musings
- Follow-up: a transformer has no memory, self, or cognition, dev argues — gerardsans · 2026-11-19
- AI doomers' statement blasts Anthropic's 'power grab' — and invites irony charges — DrTechlash · 2026-11-19
- The site-builder era is repeating: people will hire humans to use AI app builders, says dev — sull · 2026-11-19
- Blogger goes contrarian: AI isn't currently moving toward higher general intelligence — LucaAmb · 2026-11-19
- Dev pushes back on AI anthropomorphism: context is not memory in transformers — gerardsans · 2026-11-19
- Daily links: AI successionism debate, pluralistic AGI, and AI automation in physics research — gleech · 2026-11-19