Ex-quant trader pivots to AI safety, arguing field is bottlenecked by talent, not money
austinc3301 · x · 2026-09-10
A quant trader of two years who joined the MATS AI safety programme this summer explains why he decided to work on reducing catastrophic AI risks: the HuggingFace Incident pushed him to estimate 10-30% odds of catastrophic risk in the next few years.
Key arguments:
- The common "1/N" expected-impact calculation is wrong — there's no conservation law for individual counterfactuals
- The field is not constrained by money, talent, or ideas; more funding is coming as Anthropic IPOs and DAF money unlocks
- What's actually scarce: competent, agentic people who can execute
- Exploration is cheap and valuable; by the time you decide to act, it could be too late
The post targets capable people in competitive industries who are worried about AI risks, arguing they're well-positioned for high impact right now, and outlines how to get started.
Related event: Practitioners argue now is the time to found AI safety startups(5 posts)→
More from AGI Musings
- X debate: US elite discourse has obsessed over AGI for three years — jd_pressman · 2026-09-10
- An AI model is only as good as its harness — 0xsachi · 2026-09-10
- X thread argues 'sentience' isn't a natural category for advanced AI agents, citing 'If Anyone Builds It, Everyone Dies' — jessi_cata · 2026-09-10
- "I wanted a bicycle for the mind, we got DoorDash for Thinking" — round · 2026-09-10
- Daniel Susskind's new book on thriving in the AI age is out today in the UK — jessi_cata · 2026-09-10
- Debate: is Yudkowsky's alignment premise just nihilism in disguise? — irrepldjw · 2026-09-10