iamtrask: weight columns are concept detectors, and labs profit from not auditing them
iamtrask · x · 2026-11-12
iamtrask offers a theory explaining why frontier models are human-like in some ways, un-human-like in others, and full of uncontrollable emergent quirks.
They're human-like because concepts were acquired from human data; un-human-like because concepts and their interconnections are missing (cf. the reversal curse). Mechanically, each column vector in transformer weights remains a concept detector — lower-level detectors feed higher-level ones — initially semantically empty, then filled by web scrapes, purchased data, user logs, and Mechanical Turkers. The quirks persist because concept capacity is so large nobody audits it: interpretability was traded for scale 15 years ago, and frontier labs have enormous commercial incentive not to accidentally solve interpretability, since it would expose serious IP risks.
Related event: iamtrask: Transformer Weights Are Concept Detectors(3 posts)→
More from AGI Musings
- Diamandis: best diagnostician of 2030 will be a panel of arguing AI agents — PeterDiamandis · 2026-11-12
- Eric Elliott on going from $500K in tech to homeless, and an AI-era small-business outlook event — ericelliott_ · 2026-11-12
- Palantir CTO Slams Effective Altruism as Masad Questions Timing of Huang Controversy — BrettKrieger12 · 2026-11-12
- Can LLMs be trained to commit heresy? Post-training against conformity, rewarding truth by outcomes — dioscuri · 2026-11-12
- Spencer Greenberg Asks: Would You Speak to an Audience You Strongly Disagree With? — SpencrGreenberg · 2026-11-12
- Why you shouldn't just trust your agent: CMU's Niloofar Mireshghallah on privacy and AI safety — niloofar_mire · 2026-11-12