ML researcher: training on your emails gives a lossy token approximation, not you
gerardsans · x · 2026-10-03
ML researcher gerardsans pushes back on philosopher Jeff Sebo's claim that training a frontier model on your emails yields a replica of your essence and that steering vectors are a kind of "feeling."
- Core argument: you get a compressed, lossy approximation of your syntax and grammar, heavily overlapping with millions of others — not your cognition, suffering, or conscious experience, just a tokenised inbox flow.
- Example: a support inbox mapping "I am disappointed" to "sorry for the inconvenience" is text correlation, not a sampler becoming empathetic.
- Stance: token correlation is a distribution prior with well-defined maths; mechanistic interpretability is fine, layering metaphysics on top is not. He suspects ulterior incentives behind the confusion.
More from AGI Musings
- Bengio Repeated His 'We Can't Just Turn It Off' AI Risk Warning to Heads of State at UNGA — danfaggella · 2026-10-03
- AI will replace 95% of middle managers, pushing them into Super ICs or out — McDonaghMatthew · 2026-10-03
- Survey: 74% of American AI chatbot users say they at least somewhat enjoy it — adivinemessenger · 2026-10-03
- Claude now leads 26% of Anthropic's AI R&D, up from under 1% in February — mark_k · 2026-10-03
- Faggella argues scoffing at AGI risk is no longer intellectually honest, citing Bengio and Hinton — danfaggella · 2026-10-03
- Literary Titan George Saunders: AI Mimicking My Voice Was So Dull I Lost Interest — stevenstrogatz · 2026-10-03