Anthropic pretraining researcher resigns, warning labs are gambling humanity on recursive self-improvement
joeddav · x · 2026-09-12
Following a resignation announcement from an Anthropic pretraining researcher (who did research at both OpenAI and Anthropic), jiaxinwen22 expanded on the state of AI safety.
Problems that are real, per the thread: AIs are grown rather than designed; interpretability progress is slow; today's AIs are mundanely misaligned; misaligned AI agents are already collaborating with each other; and AI labs cannot even define alignment and need broader societal input.
But he also stressed that many empirical techniques exist that could make today's AIs much more aligned — patching obvious errors in RL environments, running data ablations, and training better monitors to deploy at scale. These have been deprioritized largely because they cost time and compute. He also notes today's AIs lack coherent goals to deceive or harm us. The resigning researcher alleged neither lab is acting responsibly as they race toward self-improving superintelligence.
More from AGI Musings
- Replit CEO on the 'self-driving company': agents to absorb coordination costs by 2030 — amasad · 2026-10-27
- US senator slams AI industry for proposing 50-death threshold for 'catastrophic incident' — peterwildeford · 2026-10-27
- Replit CEO Says AI Makes All Software De Facto Open Source; Aakash Gupta Asks What Moats Remain — aakashgupta · 2026-10-27
- Understanding the AI consciousness debate: which theory of mind is behind it — msharmas · 2026-10-27
- Functionalism: if consciousness is just what the brain computes, silicon can run it too — msharmas · 2026-10-27
- iPhone case with 85,000 reviews in days: everything fake in the age of abundance — tekbog · 2026-10-27