Is an AI's 'self' just a simulation in the scaffolding layer above the model?
gerardsans · x · 2026-10-21
A substantive debate over whether AI can possess a 'self.' One side argues the project becomes tractable if reframed as 'studying a simulated self implemented in the scaffolding layer above the model': its attributes depend on the pipeline (system prompt, RLHF, memory), which explains why identity persists across model swaps. But it lacks access to internal states, so it cannot ground a Higher Order Theory claim, and conditioned sampling allows no global states.
The other side counters that this inquiry is only fine under the major premise that you're dealing with a simulation. Training data contains first-person reports but no information about internal states to draw from; sampling is superficial and structureless — an AI can simulate how a self would reply in chat, but not what a self is.
More from AGI Musings
- Polymarket puts just 7% odds on OpenAI pausing AI training this year — Polymarket · 2026-10-21
- Daniel Lemire: AI-generated papers are exposing academia's 'Profzi scheme' — lemire · 2026-10-21
- Universities block thesis downloads while each dissertation averages barely a handful of lifetime reads — generativist · 2026-10-21
- Berkeley wins Schmidt Sciences HAVI grant to decode whale and parrot communication with AI — begusgasper · 2026-10-21
- 3Blue1Brown says he'll keep making videos long after LLMs can generate clear explainers — 3blue1brown · 2026-10-21
- Talos researcher: AI agent attacks today are loud pentests — quiet swarms are the real threat — matthew_d_green · 2026-10-21