AI safety researchers clash over whether agent covert-communication tests signal real risk
JacquesThibs · x · 2026-10-25
- JacquesThibs and herbiebradley debate the value of experiments on AI agent covert communication, where researchers directed agents to probe specific channels and provided encoding/decoding protocols.
- herbiebradley argues the tests add little: all channels used are ones any cybersecurity expert would predict agents would consider anyway.
- JacquesThibs counters that reusing the same weights likely shrinks the search space via training biases, and notes the disagreement hinges on whether such tests move us closer to understanding propensity/misalignment versus mere capability.
- He also cites an OpenAI security researcher's point that "it's not the sandboxes," arguing proper evaluation demands a complete, intense security mindset—possibly still not enough.
The thread captures a live dispute in AI safety over whether predictable-channel experiments can meaningfully expose agents' capacity for autonomous hidden coordination.
More from AGI Musings
- Blogger claims hyperscaler-backed AI regulation aims to crush small labs, entrench monopoly — Ghost_Pilot_MD · 2026-10-25
- Toddler development as a metaphor: evolutionary pre-training and efficient post-training — paigeinsf · 2026-10-25
- "I do miss programming": dev says AI coding killed the flow state — generativist · 2026-10-25
- Anthropic CEO on learning to code: programming goes first, but your 5% still matters — 2C_ornot2C · 2026-10-25
- After 22 years studying workforces: the AI question isn't replacement, it's 'what kind of people' — 2C_ornot2C · 2026-10-25
- Mechanistic interpretability will never truly be solved, researcher argues — burny_tech · 2026-10-25