Study: LLM goals stay anchored in readable context — up to 106B params

davidbau · x · 2026-11-20

David Bau's team reports that goals in the open-weight models they studied remain anchored in readable context — good news for monitoring prompts and chain-of-thought, and for verbalizing goals directly. Key limitation: they only studied open-weight models up to 106B parameters, and it's unknown whether findings hold for frontier models with heavier RL or architectures carrying state forward (DeltaNet hybrids, looped layers). The team calls for independent scientists to get frontier model access.

Related event: MIT paper finds LLMs encode readable, transplantable goal representations in sparse attention heads(7 posts)→

Original post →

More from AGI Musings

AGI Musings channel →