Debating the mathematician letter: can AI search be indexed for serendipity instead of goodharted?
curious_vii · x · 2026-09-12
A take on the mathematician letter: labs may be goodharting math exploration, and blunt automated search could miss serendipitous insights a human would stumble on. The author proposes indexing digital exhaust with proxies for interestingness/surprise (echoing Schmidhuber) for human review, but concludes the letter essentially reads as "don't solve our puzzles."
More from AGI Musings
- Bengio lays out analysis of misaligned agents; sebkrier pitches adversarial multi-agent checks — sebkrier · 2026-09-12
- Doomer take: mathematicians have one year before open AI models sweep Millennium Prizes — teortaxesTex · 2026-09-12
- The Real AI Doom Scenario Isn't Terminator, It's a Mental WALL-E — eigenhector · 2026-09-12
- Maybe Cloud Atlas Got the AI Apocalypse Right, Says Thread — eigenhector · 2026-09-12
- The final UI is clear: language in, intelligence, language or generated UI out — signulll · 2026-09-12
- AI Doom Was Never Terminator, Just People Outsourcing Thinking to Robots — eigenhector · 2026-09-12