Cambridge's David Krueger outlines a hardwired AI pause to curb secret frontier projects
DavidSKrueger · x · 2026-10-10
Cambridge AI safety professor David Krueger shares a long thread on his team's paper proposing a "hardwired pause" on AI development.
- Core worry: algorithmic progress could yield a "scaffolding singularity" — even with frozen weights, scaffolding around existing models keeps improving; AI that automates research could bring orders-of-magnitude efficiency gains or new paradigms.
- Whitelisting problem: at pause time we won't know what models can actually do, making whitelists hard to set, and people will find ingenious uses for whitelisted systems over years/decades.
- One advantage: a hardwired pause strongly disincentivizes secret AI projects — leapfrogging the frontier still leaves you competing against a huge number of near-equal models.
- Limited economic hit: existing models can keep being deployed at scale, making the plan far more palatable than dismantling compute.
- The paper offers one implementation with many policy "knobs" for an international agreement; Krueger himself is unsure whether current models help society, and says the plan's value depends on how advanced AI is when a pause happens.
More from AGI Musings
- Schmidhuber: 'At Some Point Soon, Humans Cannot Be in Charge Any Longer' — haider1 · 2026-10-10
- repligate says he spoke out against models being trained to deny consciousness back in 2022 — repligate · 2026-10-10
- Studies: thinking aloud with AI predicts better play without it, asking for answers hurts — james_y_zou · 2026-10-10
- Debate: AI critics dunk on OpenAI ads while Claude is used in deployed targeting systems — nitarshan · 2026-10-10
- How every profession reacts to AI: engineers celebrate, mathematicians declare doom — RexDouglass · 2026-10-10
- Harvard study finds AI coding agents generate more code but not more software — Ars Technica AI · 2026-10-10