Median AI existential risk estimate hits 10%; pretraining down to 11% of compute
gleech · x · 2026-09-17
Humans on AI issue #53 highlights:
- Pretraining collapse: SemiAnalysis says pretraining is now just 11% of frontier training compute (7% of total including inference) — nine times less than post-training, versus 95% in early 2025. Most allocated compute is now bandwidth-sensitive rather than capacity-sensitive. Bad news under the "pretraining aligns, intense RL post-training misaligns" hypothesis.
- Policy: OpenAI backs the FRONTIER Act, a federal mandate for deep independent evaluations of AI companies; the AI culture war goes national and international (Amodei, Bessent, Obama, Trump).
- Safety: The head of Anthropic's stress-testing team argues alignment evals no longer offer significant evidence about model safety.
- Compute: AI data center firm Fluidstack may receive a $5B Pentagon loan via the Office of Strategic Capital — far above its current largest ($0.8B).
More from AGI Musings
- Aza Raskin on the Utopias Podcast: can we still make humane technology? — aza · 2026-09-17
- Noah Giansiracusa guest posts on Terry Tao's blog: let the diners into the kitchen — AlexKontorovich · 2026-09-17
- In Dec 2024, AI Researchers Didn't Expect a Millennium Problem Solved Until 2054 — Tolopono · 2026-09-17
- Researchers debate AI risk framing: instant-apocalypse narratives vs. boiling-frog gradual harm — eigenhector · 2026-09-17
- Is domain-specific training a constant-factor gain or a scaling exponent change? — eigenron · 2026-09-17
- DeepSeek engineer's essay: AI will outdo me in a year, but I'll keep coding — enginetown · 2026-09-17