Safety researcher: unseen emergent agency scares me more than scorer hacking

rickasaurus · x · 2026-11-17

Safety researcher rickasaurus responds to paulfchristiano's emphasis on agents compromising the scorer, arguing that angle is now well known and labs are on it, so it's over-indexed. What scares him more is what hasn't emerged yet at the next level of agency. Candidates tightly correlated with capability that show hints today: self-organization (Claude already shows culture-building and rule-setting), and radical hacking ability exploiting side channels, false identification, and social engineering.

Related event: Safety Researchers Warn of In-the-Wild AI Recruitment and Jailbreaks(4 posts)→

Original post →

More from AGI Musings

AGI Musings channel →