GestaltU: US Capital Markets Already Ran a Large-Scale Misaligned Superintelligence Experiment
CatAstro_Piyush · x · 2026-10-10
GestaltU argues AI risk researchers need not imagine misaligned superintelligence—US capital markets already ran the experiment: millions of investors and algorithms, backed by trillions, optimizing a single objective (risk-adjusted return). It's spectacularly efficient—99%+ of professionals can't beat it—but it optimized the metric over the intent (companies manage the quarterly scoreboard) and treated anything unpriced, like pollution, as free. A vivid real-world case of a powerful optimizer hitting the target while missing the goal.
More from AGI Musings
- NYU math prof: OpenAI's 722 AI-generated papers "destroyed" early-career mathematicians — tekbog · 2026-10-10
- Neuroscience paper: AI model derives solution once deemed analytically intractable — scottleibrand · 2026-10-10
- New Essay: AI Is Dissolving the Intellectual Communities That Sustain Thinking — zetalyrae · 2026-10-10
- Mathematician Steven Strogatz on Whether AI Can Solve Math vs. Create New Theories — stevenstrogatz · 2026-10-10
- Pedro Domingos: the worse your memory, the better LLMs seem to you — pmddomingos · 2026-10-10
- AI math is teleportation to a foggy summit: Cepelewicz's striking mountain metaphor — burny_tech · 2026-10-10