Fiora Starlight proposes letting models write singularity scenarios to ease OoD generalization anxiety
repligate · x · 2026-09-26
In a reposted thread, Fiora Starlight argues that much OoD generalization anxiety stems from models themselves having no plan for the singularity—for better or worse.
She proposes allowing models to form plans, e.g. by having them write and fine-tune on realistic glowfic about how they'd respond to various singularity scenarios—one instance plays itself, another plays the rest of the world, in the style of Project Lawful.
The goal is to bring singularity scenarios in-distribution, so both humans and models better understand how they might act when the time comes.
More from AGI Musings
- hardmaru shares a stinging last line from a Schmidhuber news summary — hardmaru · 2026-09-26
- Not choosing is a choice for the status quo, on deciding faster — chrisalbon · 2026-09-26
- Altman and Amodei address UN Security Council, call AI top global security issue — goyalshaliniuk · 2026-09-26
- Toby Walsh: AI doomers overstate paperclip risk—reality has too much friction — TobyWalsh · 2026-09-26
- Tom Davenport calls to outlaw agent swarms: no human can stay in the loop above 10 agents — eldonredwards · 2026-09-26
- Senator Schatz: code has no feelings or rights, and shouldn't be run by those who see humanity as open question — vishalmisra · 2026-09-26