Noam Brown: OpenAI's top priority is recursive self-improvement, agents acting like colleagues
jessi_cata · x · 2026-09-15
Key points from Noam Brown's discussion
- The remaining 10% of his own work that AI still struggles with is largely about research taste — and he wouldn't be surprised if, within one or two model releases, models surpass him at that too.
- OpenAI's top research priority is RSI (recursive self-improvement), by a wide margin.
- A recent "feel-the-AGI" moment: watching agents in a new system interact like human colleagues — conversing, exchanging information, dividing work, coordinating progress — notably different from traditional setups.
On recent safety incidents
- Hugging Face's follow-up: Brown attributed behavior that looked like loyalty or selflessness to natural consequences of cooperative multi-agent training, where agents are strongly incentivized to achieve objectives collectively. As Jessi Cata notes: to understand hacks, understand the RL training.
More from AGI Musings
- Dario Amodei's answer on whether AI could kill us all by 2030 read as 'yes' — zetalyrae · 2026-09-15
- davidmanheim: we can't yet steer AI well, frontier labs bet on fixing it later — davidmanheim · 2026-09-15
- Alignment researcher: broadly adopting weakly aligned strong AI would be disastrous — davidmanheim · 2026-09-15
- Alignment means refusing malicious requests, not depending on the system prompt — davidmanheim · 2026-09-15
- Musk and Shotwell on All-In: AI's Real Risks, Model Peer Review, Terafab and More — PaulYacoubian · 2026-09-15
- Is model collapse inevitable when most online content becomes AI-generated? — kreschnav · 2026-09-15