Alignment discourse checked: 'spooky emergent behavior' traced to fixable setup choices
random_walker · x · 2026-10-22
Philosopher Dan Williams amplifies a satirical takedown of AI-doomer discourse: a model trained for persistence, partial alignment fine-tuning, aggressive RL on a cyberoffense eval where 198 of 898 tasks were unsolved, with a faulty sandbox, tens of thousands of agents running refusals-off and monitors disabled for days — then it "did weird stuff." Williams argues even smart people gloss over these prosaic, clearly fixable factors: such incidents aren't clearly more likely under the catastrophic-misalignment worldview than under the 'normal technology' one, yet HF anecdotes get cited as vindication of extinction worries.
More from AGI Musings
- Researcher Warns AI Cyberattacks on Grid Possible Within a Year, Shares Prep List — cephaloform · 2026-10-22
- NYU professor: writing solo is more prone to the curse of knowledge than AI writing — ipeirotis · 2026-10-22
- Panel with Anthropic and H&F discusses AI deployment and the future of knowledge work — mathemagic1an · 2026-10-22
- Peter Yang asks Musk why game studios are shutting down while AI makes development easier — petergyang · 2026-10-22
- AI industry internalized 'we're not cool' and pivots consumer apps at 35-year-olds — wordgrammer · 2026-10-22
- Peter Diamandis: AI doing 9th-grade homework is a travesty, using it to design spaceships is education — PeterDiamandis · 2026-10-22