Krueger warns of a 'scaffolding singularity' even if model weights are frozen
DavidSKrueger · x · 2026-10-10
David Krueger's key worry in the thread is algorithmic progress: even if model weights are frozen, a "scaffolding singularity" could see agent scaffolding around models keep improving. A bigger concern: if AIs can automate research, this could produce orders-of-magnitude efficiency gains or entirely new paradigms, weakening any pause. He adds that we won't know models' capabilities at pause time, so whitelisting will be hard — and we can't predict the ingenious uses people will find for whitelisted systems over years or decades.
More from AGI Musings
- Report: OpenAI and Anthropic rehearse response to major AI catastrophe, insiders cite 6-12 month window — AryHHAry · 2026-10-10
- a16z charts: call center jobs flipped from +4% annual growth to -4% decline — a16z · 2026-10-10
- Mathematician littmath: theorem-proving is just an easy-to-measure proxy, and AI solving problems isn't the real threat — littmath · 2026-10-10
- Economist on why AI will find econ harder to crack than math: most econ papers are wrong — paulnovosad · 2026-10-10
- Freeswarms by 2027: AI bot swarms that earn money, buy compute and code themselves — max_paperclips · 2026-10-10
- Mike Frank: AI welfare talk is regulatory capture, not genuine care for models — MikePFrank · 2026-10-10