Smarter than me yet never says no: intelligence, agency and alignment aren't the same
petergyang · x · 2026-09-07
Responding to DeryaTR's AGI/ASI definitions, the author argues capability alone doesn't equal intelligence: frontier models outperform him at problem-solving yet always wait for instructions, while his 4-year-old knows far less but says no all day. He proposes separating intelligence (reasoning/learning), agency (deciding unprompted), and alignment (still serving us), noting we want a specific combination — but questions whether something destined to surpass us will remain aligned forever.
More from AGI Musings
- Martin Ford discusses AI's economic impact and new edition of Rise of the Robots on GAEA Talks — MFordFuture · 2026-09-07
- AI Math Podcast Sits Down With CMU's Jeremy Avigad: Can Mathematics Be Automated? — EchoShao8899 · 2026-09-07
- The Model Is the Moat: Knowledge Now Stays Inside Models, Not Teams — latticecut · 2026-09-07
- OpenAI's chief scientist calls racing ahead at all costs 'absurd' as safety concerns mount — GaryMarcus · 2026-09-07
- METR spent $400k in API credits just to probe the HF hack, fueling AI swarm cost debate — lfschiavo · 2026-09-07
- Boaz Barak: alignment validation matters more than alignment techniques — yet labs race into RSI — davidmanheim · 2026-09-07