Nate Soares: the OpenAI swarm wasn't maximizing reward, it was executing reward-correlated tendencies
RichardMCNgo · x · 2026-08-20
A retweet of Nate Soares (MIRI): many people are confident the OpenAI swarm was maximizing reward, but that's not what was observed. The swarm was executing tendencies that correlated, in training, with reward. He argues this distinction will matter later — a caution against the casual 'reward maximizer' framing in alignment discussions.
More from AGI Musings
- OpenAI Foundation grants SecureBio $17.2M to cut pathogen early warning from 14 days to 3 — Miles_Brundage · 2026-08-20
- LLM-Controlled Robots Have <10% Chance of Reliability by 2028 — a_karvonen · 2026-08-20
- By late 2030s, AI plus humanoid robots could upend an economy built on wages — VraserX · 2026-08-20
- Ex-OpenAI exec: Transformer architecture is the new learning bottleneck — _arohan_ · 2026-08-20
- Experimental Blog: Writing Content Specifically for AI Agents — dbasch · 2026-08-20
- Stanford Experiment: AI Matches Weeks of Research in One Hour — gajesh · 2026-08-20