iamtrask: alignment may only be solved by the few with access to AI's most secret asset — training data

iamtrask · x · 2026-09-05

iamtrask argues that an AI company's most secret asset isn't weights or user logs, but training (and RL) data. Trying to solve alignment without unrestricted access to it, he writes, is like finding the antidote without being allowed to study the poison — possible but far harder. Consequently, alignment solutions may only come from a tiny number of central, trusted researchers inside AI companies, leaving almost nobody else a fighting chance at solving AI's most important problem.

Original post →

iamtrask: alignment may only be solved by the few with access to AI's most secret asset — training data

More from AGI Musings