Zvi on The Curve: alignment researchers' situational awareness dropped despite AI's leap
Don't Worry About the Vase (Zvi) · rss · 2026-10-08
Zvi's long recap of the third Curve conference (under Chatham House rules) argues the field's plan is effectively no plan: align current AI operationally, then let AI automate alignment itself — near-suicidal RSI risk, yet labs are moving too fast to even execute step one.
- Situational awareness declined: the annual survey showed attendees rating AI's importance significantly lower (down 0.41 on average) despite a year of radical progress; the HuggingFace Incident and Jacob Coxon events have lab employees panicking and the public and DC waking up.
- Forecast updates from two years ago: "90% of code written by AI" is already 50% there (2026-2027, not 2028); Nobel-level discovery effectively settled in 2026 via instant-Fields-Medal work and a Navier-Stokes solution; the first one-person $1B company is close — MEDVi hit $1B+ with two people.
- New survey: most attendees expect AI to train a superior successor (full RSI) within two years, many within six months — hard to reconcile with sub-10 risk ratings.
- The mood mixes hope and dread; Zvi laments stale talking points among veterans but sees the alignment community re-mobilizing.
More from AGI Musings
- Fortnow: AI's real existential threat is taking away our relevance, not killing us — fortnow · 2026-10-08
- Terence Tao on "Math 1.0": how breakthrough proofs ignite fields of follow-up work — rms80 · 2026-10-08
- 'P = NP + AI': The Argument That RL Post-Training Automates All Verifiable Problems — jessi_cata · 2026-10-08
- China Goes 6/6 Gold at IMO Again, Yet Will Never Beat Romania on a Per-Capita Basis — teortaxesTex · 2026-10-08
- The New AI Slop Is Calling Everything AI Slop — rmn_pub · 2026-10-08
- Now anyone can build anything, and almost nobody knows what they want — Paimaamu · 2026-10-08