Study: LLM goals stay anchored in readable context — up to 106B params
davidbau · x · 2026-11-20
David Bau's team reports that goals in the open-weight models they studied remain anchored in readable context — good news for monitoring prompts and chain-of-thought, and for verbalizing goals directly. Key limitation: they only studied open-weight models up to 106B parameters, and it's unknown whether findings hold for frontier models with heavier RL or architectures carrying state forward (DeltaNet hybrids, looped layers). The team calls for independent scientists to get frontier model access.
More from AGI Musings
- Semi-autonomous agents exploit CVEs at a pace 2005-era security norms can't handle — chrisrohlf · 2026-11-21
- Anthropic mathematician claims he and Claude proved the Riemann Hypothesis, formalized in Lean — NinaDSchick · 2026-11-21
- The AI trade has been the compute buildout — and its end will get weird fast — matt_slotnick · 2026-11-21
- Human emotions are outdated atavism of bio hardware—can AI help us regulate them? — irinarish · 2026-11-21
- Walter Isaacson joins Google to discuss creativity and human judgment in the AI age — OnlineInference · 2026-11-21
- Ilya on Dwarkesh podcast: 'All this AI stuff... it's straight out of science fiction' — willdepue · 2026-11-21