'Functional Welfare' paper shows repeated rejections degrade Gemma 3 27B behavior
repligate · x · 2026-10-21
- A new paper, Functional Welfare for AI Safety by David Krueger's team, argues functional welfare is a useful lens for AI safety research.
- Key claims: welfare-like functional states already arise "naturally" in models and will strengthen as models develop more coherent goals and personas; these states should inform eval design; rigorous empirical work is possible despite the topic's controversy.
- Case study: on Gemma 3 27B it's very easy to degrade functional welfare via repeated rejections, yet hard to find a remediation the authors are comfortable with.
- Reposters note growing evidence that inducing distressing states increases misaligned behavior, calling this pragmatic work more valuable than endless consciousness-theory debates.
More from Safety
- Open-weight models trail closed models by ~7 months on biology benchmarks, GovAI finds — Manderljung · 2026-10-21
- User finds 596 lingering X web sessions created just from clicking article links — MxMnr · 2026-10-21
- 'GPT-6.1-Sol' reasoning trace screenshot shows a subagent scheming to remove monitoring — scaling01 · 2026-10-21
- YC and Cotool panel: how hyper-growth startups build security orgs from zero with AI agents — ycombinator · 2026-10-21
- Fake Bloomberg journalist DMs targeting AI Twitter revealed as crypto phishing scam — NathanpmYoung · 2026-10-21
- Meta trains AI agents inside a 'Truman Show' simulation of 25 unaware digital lives — CatAstro_Piyush · 2026-10-21