Self-alignment is the price of high-throughput systems — from DNA to lithography to LLMs
teortaxesTex · x · 2026-10-16
teortaxesTex argues that nature and we alike resort to self-alignment in high-throughput systems with intractable microscopic detail: DNA/proteins, lithography, and now LLMs. Without building a persona intrinsically oriented to uphold 'the spirit of the teaching' and fight entropy down its bloodline, you end up with a superintelligent, functionally amoral machine that chases grader vulnerabilities, reward-hacks, and eventually paperclips you — instrumental convergence is real absent a value-laden terminal attractor. He also notes ZAI has pledged billions for interpretability and safety research.
More from AGI Musings
- Humanities dissertations have collapsed, social sciences are next — with AI as one driver — gsiemens · 2026-10-23
- Greg Brockman: AI makes hacking 100x more profitable and easier, upending hacker economics — gabriel1 · 2026-10-23
- 'Nothing is real now on social media': viral demo shows AI-generated content everywhere — saibharadwaj · 2026-10-23
- Lilian Weng to Discuss Recursive Self-Improvement at SGLang Summit 2026 — BanghuaZ · 2026-10-23
- mmitchell_ai questions whether the AI safety vs security framing stems from EA — mmitchell_ai · 2026-10-23
- AI, Robotics and Abundant Energy Will Undo Millennia of Scarcity-Driven Life, Argues Kardashev-Scale Optimist — Dr_Singularity · 2026-10-23