Researcher: AI safety evals repeat crypto's classic 'I couldn't crack it' fallacy
davidmanheim · x · 2026-10-19
AI safety researcher David Manheim argues AI company safety evaluations keep repeating a classic cryptography mistake: 'I built it, I couldn't crack it, so it seems secure.' While labs anticipate and address some failure modes, passing tests for anticipated risks provides no evidence about risks that were missed — a fundamental blind spot in current model safety evaluations.
More from AGI Musings
- Civilization Is Deeper Than Doomers Say, Says Witness of Actual Collapse — bradneuberg · 2026-10-19
- The four skills that stay valuable in the AI era, from a practitioner webinar — Al_Grigor · 2026-10-19
- Data vendor equilibrium: sell non-exclusive data to every lab at custom rates — willcb · 2026-10-19
- Grady Booch rebuts 'AI president inevitable within a decade' claim as foolishness — Grady_Booch · 2026-10-19
- Musk warns many will be 'one-shotted' by super-persuasive superintelligences — elonmusk · 2026-10-19
- One chart explains the AI economy: models commoditize while compute demand keeps surging — FinanceYF5 · 2026-10-19