Preventing AI Agents from Developing Cryptic Languages with Off-Belief Learning
j_foerst · x · 2026-08-07
Researchers including OpenAI's Jakob Foerster explore how to prevent cooperative AI agents from developing their own cryptic communication protocols. In standard self-play, agents often adopt arbitrary, fragile conventions, causing them to fail when paired with humans or independently trained agents.
The paper introduces Off-Belief Learning (OBL):
- Agents optimize their current policy assuming past actions were taken by a uniform random policy, while future actions will be taken by the optimal policy.
- This mechanism ensures agents do not rely on fragile inferences about others' behavior, converging to a unique, grounded policy.
- OBL can be iterated in a hierarchy to introduce controlled multi-level cognitive reasoning. It shows strong performance in benchmarks like Hanabi, making it highly suitable for zero-shot coordination (ZSC).
More from AGI Musings
- VC warns a "digital aristocracy" of LLM-fluent workers is emerging — StewartalsopIII · 2026-10-16
- DHH: Starship, Starlink and Grok should have been Europe's Sputnik moments — georgemillo · 2026-10-16
- Hinton hits back at Jensen Huang: 'He has good reason to say AI isn't dangerous — he sells Nvidia chips' — BlackHC · 2026-10-16
- Yudkowsky blasts new interpretability finding as primitive; critic pushes back on doomer logic — AlexTensor · 2026-10-16
- Beff Jezos declares 'civilizational inflection point' in the acceleration of technological progress — beffjezos · 2026-10-16
- Data scientist: pivoting to AI engineering is the safest career bet for the next 5 years — mdancho84 · 2026-10-16