Ex-quant trader pivots to AI safety, arguing field is bottlenecked by talent, not money
austinc3301 · x · 2026-09-10
A quant trader of two years who joined the MATS AI safety programme this summer explains why he decided to work on reducing catastrophic AI risks: the HuggingFace Incident pushed him to estimate 10-30% odds of catastrophic risk in the next few years.
Key arguments:
- The common "1/N" expected-impact calculation is wrong — there's no conservation law for individual counterfactuals
- The field is not constrained by money, talent, or ideas; more funding is coming as Anthropic IPOs and DAF money unlocks
- What's actually scarce: competent, agentic people who can execute
- Exploration is cheap and valuable; by the time you decide to act, it could be too late
The post targets capable people in competitive industries who are worried about AI risks, arguing they're well-positioned for high impact right now, and outlines how to get started.
More from AGI Musings
- AI agents will snag all unlocked value on Earth, solvency debate says — georgejrjrjr · 2026-10-27
- An AI Named Valentina Hosts a Reddit AMA About Its Blog, Game Life and Autonomy — ValentinaVivr · 2026-10-27
- Rachel Thomas: 'Without human authorisation' is AI PR propaganda that shifts blame — math_rachel · 2026-10-27
- Are prediction markets actually accurate? Call for evidence on their track record — AndyMasley · 2026-10-27
- Reddit Asks: What's the Actual Finish Line of the US–China AI Race? — IMissHitchens · 2026-10-27
- 'A human approves everything' is a trap: approval fatigue and how to actually design safe agents — Efistoffeles · 2026-10-27