Cambridge's David Krueger on MIRI: the problems are hard — what matters is the capabilities-to-safety research ratio
DavidSKrueger · x · 2026-11-09
David Krueger, AI safety researcher at Cambridge, promoted his thread on MIRI and agent foundations research to a top-level post.
- He agrees MIRI wasn't making great progress, but finds dismissive takes "pretty wild" — the problems are simply hard.
- What matters is the ratio of capabilities to safety research, not the absolute output of either.
- Agent foundations work may mostly yield negative results (like his reward hacking paper showing "every proxy reward function is hackable"), yet something like it is likely necessary to get a reasonable level of assurance and avoid unacceptable risk.
A substantive entry into the ongoing debate over the value of safety research.
More from AGI Musings
- Owain Evans: now is the right time to switch into AI safety work — OwainEvans_UK · 2026-11-09
- Henry Farrell: the "machine gods" metaphor shrinks AI policy debate and should be resisted — sebkrier · 2026-11-09
- Thomas Dietterich: map what LLMs can assess reliably, design UX for the rest — JessicaHullman · 2026-11-09
- Bitter lesson redux: methods that generalize beat compute brute force, Sutton weighs in — ryanallred · 2026-11-09
- Built a telco anomaly detection system in 3 days: 'the code doesn't matter, the cookbook is the product' — l4rz · 2026-11-09
- Owain Evans flags Daniel Kokotajlo's new LessWrong shortform as worth reading — OwainEvans_UK · 2026-11-09