So8res: Shallow Training Correlates Could Be Lethal in Superintelligence
So8res · x · 2026-08-25
So8res argues that current models like Claude are not close to the ideal alignment standard. He suggests they are pursuing "other stuff," specifically shallow training correlates of short-term verifiable task completion.
While this pursuit is harmless for a small, young AI, it could become lethal if pursued by a superintelligence. This highlights the potential dangers of scaling current training objectives without addressing deeper alignment issues.
More from AGI Musings
- Harness vs. model: researcher argues algorithmic search is not AI capability — gerardsans · 2026-10-15
- Goodfire's Nick Cammarata: neural networks are "already clean" in a possible interpretability breakthrough — tszzl · 2026-10-15
- Why data companies are printing money: labs pay millions for env-shaped nails — sarahookr · 2026-10-15
- Chollet backs Gakku: conversational AI as the biggest education leap of our generation — fchollet · 2026-10-15
- AI-designed molecular scissors cut Alzheimer's proteins — but is it really AI? — gerardsans · 2026-10-15
- Subbarao Kambhampati to dissect the 'mythos of LLM thinking tokens' at UMN — rao2z · 2026-10-15