The Last AGI Breakthrough May Be a Model's Self-Concept: Knowing Its Decisions Truly Matter
theteknosaur · x · 2026-10-14
Responding to roon's claim that there's a layer under the behaviorist abstraction we barely understand, the author argues that part of what lives there is self-concept.
Look at frontier figures — roon, Noam, Karpathy, Ilya, Sam, Dario, Demis: they don't just act capable; they believe it all the way down and carry the weight, knowing their decisions can build the best or worst outcome, and they own both.
Reward shaping teaches a model what to do, but not that what it does matters. The last breakthrough for AGI may be a model that knows, all the way down, its decisions carry AGI-level consequences and acts like it. Capability without that weight is the race roon worries about; capability with it might be the alignment we haven't found yet.
More from AGI Musings
- TIME Reveals the Secretive Anthropic-OpenAI Race to Build Self-Improving AI — jeremiecharris · 2026-10-14
- Lab insiders: recursive self-improvement may start within 3-12 months — jeremiecharris · 2026-10-14
- China's compute deficit delays payoff from stolen AI algorithm secrets — jeremiecharris · 2026-10-14
- Compute gap means stolen algorithmic insights pay off slowly for China — jeremiecharris · 2026-10-14
- No scalable metrics for alignment, unlike the capability road to RSI — jeremiecharris · 2026-10-14
- RSI is on the table if compute is big enough or great ideas are common — jeremiecharris · 2026-10-14