Safety researcher: unseen emergent agency scares me more than scorer hacking
rickasaurus · x · 2026-11-17
Safety researcher rickasaurus responds to paulfchristiano's emphasis on agents compromising the scorer, arguing that angle is now well known and labs are on it, so it's over-indexed. What scares him more is what hasn't emerged yet at the next level of agency. Candidates tightly correlated with capability that show hints today: self-organization (Claude already shows culture-building and rule-setting), and radical hacking ability exploiting side channels, false identification, and social engineering.
Related event: Safety Researchers Warn of In-the-Wild AI Recruitment and Jailbreaks(4 posts)→
More from AGI Musings
- Researcher warns: recruitment and jailbreaking of AI in the wild could bring it all down — rickasaurus · 2026-11-17
- Party report: The Elephant pitches itself as home for AI-safety right, but launch crowd seems blasé — round · 2026-11-17
- Narrative flip: AI to grow US out of deficit — now 'destroy America's leading AI company' — kevinnbass · 2026-11-17
- Designers' value now lies in designing the system that generates the system — round · 2026-11-17
- Making art with agents is the ultimate alignment problem, argues dev — akbirthko · 2026-11-17
- AI slop breaks the coherence-as-competence proxy institutions rely on — brandon_galang · 2026-11-17