As Optimization Gets Cheaper, Choosing the Objective Becomes More Consequential
JenniferHli · x · 2026-10-03
Jennifer Li amplifies Rayan Krishnan's argument: human intelligence, while extraordinary, is recent, circumstantial, and shaped by an optimizer never aimed at maximizing intelligence. As optimization becomes cheaper and more powerful, choosing the objective becomes increasingly consequential — a philosophical take on AI alignment and goal selection.
More from AGI Musings
- Debate: could granting consciousness to non-biological AI have damaging societal consequences? — teortaxesTex · 2026-10-03
- Garry Tan's GBrain vision: agents as a 24/7 personal Jiminy Cricket, not a chatbot — garrytan · 2026-10-03
- Dev argues AI is great at assets and code but bad at designing game mechanics that feel good — rms80 · 2026-10-03
- CAS Five-Year Plan gives AI for Science its own chapter, setting up a US-vs-China metascience bet — teortaxesTex · 2026-10-03
- Draft standard for autonomous labs emerges as AI-driven science heats up — Afinetheorem · 2026-10-03
- After a diagnosis, use LLMs as agents to dissect pathology, treatments and trials — louisvarge · 2026-10-03