Microsoft's Chief Scientist on King Charles's AI Summit: Human Capacity Is Part of the Risk Surface
jteevan · x · 2026-10-14
Microsoft Chief Scientist Jaime Teevan reflects on the AI Summit convened by King Charles at Dumfries House, extending into how AI shapes human judgment.
Three summit topics:
- Risk: Views ranged from "just an engineering problem" to catastrophic risk; key mechanisms were misuse and misalignment, with more extreme-misalignment talk than she expected. What she found missing: human capability itself is part of the risk surface—the question isn't only who stays in control, but what kind of people exercise it. Control can erode long before anything takes it; AI can make a person faster while making their organization stupider. What's needed: better measurement—identify failure modes and measure intermediate outcomes rather than pulling catastrophe probabilities from thin air.
- Model: Discussion centered on recursive self-improvement (RSI) producing deeply misaligned capability. She notes improvement depends on environment—games have clear success signals; real-world signals come from people and human systems.
- Core thesis: humans shape AI, AI shapes humans, and those humans shape future AI. She shared an essay with the Vatican shortly before Magnifica Humanitas was published.
More from AGI Musings
- Web browsing is splitting: ridiculous for utility, amazing for entertainment — signulll · 2026-10-14
- Open models give users an exit from a few companies deciding what tools exist and say — neil_chilson · 2026-10-14
- Anthropic bans cruel treatment of Claude; critics warn of humanizing AI — AlexTensor · 2026-10-14
- Hollywood sold the dream of anyone making movies—until AI made it real — VraserX · 2026-10-14
- Google AI diagnosis in Lancet: conversation alone nails final diagnosis in 90% of cases — zakkohane · 2026-10-14
- Continual learning targets out-of-distribution intelligence, not infinite memory — akbirthko · 2026-10-14