Emmett Shear's Softmax essay argues trust is a property of relationships, not models
Hidenori8Tanaka · x · 2026-10-28
Twitch co-founder and former OpenAI interim CEO Emmett Shear, with cofounder David Bloomin, published "Origins of Trust" on Softmax addressing where trust in AI ultimately comes from:
- Theory 1: Trust as a property of the model — test behavior, read chain-of-thought, probe activations, build a theory of generalization before deploying; keep it under non-subvertible oversight until then.
- Theory 2 (the authors'): Trust between minds has never worked that way — people trust those they grew up with, depend on, and have track records with; human-AI trust will work the same way, making alignment an ongoing process a population of humans and agents keeps doing, not a one-time check.
The essay lays out both theories' assumptions and implications. A substantive alignment argument from a major industry figure.
More from AGI Musings
- Rachel Thomas points to classic paper on articulating the value of not building tech — math_rachel · 2026-10-28
- Hypothetical $2T Anthropic IPO framed as channeling $100B+ a year into the real economy — sanjaykalra · 2026-10-28
- Biotech CEO: drug discovery will lag in the AI revolution, not lead it — vaibhavbetter · 2026-10-28
- AI cyber-wreckage noise far outstrips actual damage so far, argues new analysis — timigod · 2026-10-28
- 'Enjoy the low-polygon era': F-117 meme captures today's AI constraints — willcb · 2026-10-28
- 40 Startups Building Payments for AI Agents, Mapped in 7 Categories — FinanceYF5 · 2026-10-28