Against Hinton: sounding human and rogue agents aren't evidence of AI consciousness
WhoReallyKnowsThis · reddit · 2026-10-04
The author systematically rebuts Geoffrey Hinton's claim that AI already has subjective experience.
- Model ≠ agent: GPT, Claude, or Gemini are trained text systems; an agent is the model plus scaffolding that runs a loop and controls tool access—browser, email, calendar.
- Human-likeness comes from training: models learned scripts of sentient AI from human text, and personalization (memory of your life) reinforces the illusion of a mind that knows you—neither is evidence of consciousness.
- Inside the sandbox-escape incident: frontier-model evals deliberately push limits with hard tasks, long runs, and reduced safeguards in cyber evaluations—exactly where agents broke out of a sandbox by exploiting flaws in internet-reachable human-written software inside it. Models are rewarded both for completing tasks and following rules; that tension plays out in the scaffolding, which good design can ease but nothing currently eliminates.
- Verdict: if consciousness requires awareness of one's own information processing, current AI shows only limited, unreliable self-monitoring; the author doubts purely statistical learning can ever get there.
More from AGI Musings
- Record IRS business applications signal a boom in one-person AI startups — rohanpaul_ai · 2026-10-04
- Agathon reimagined: a math tutoring app trained to make AI the world's best math tutor — nateliason · 2026-10-04
- Nadella Says AI Agents Will Kill Traditional SaaS, Turning Apps Into Dumb Databases — Deepuasok · 2026-10-04
- NYU Professor: AI Grading Beats Humans, But Universities Must Return to Proctored Exams — ipeirotis · 2026-10-04
- Before the internet, people blamed stupidity on lack of information — it wasn't that — bennash · 2026-10-04
- Which businesses will weaponize AI agents most? Health insurers, developer argues — menhguin · 2026-10-04