John Schulman breaks down three ways AI firms train on user data
anshulkundaje · x · 2026-09-15
John Schulman explains that 'training on user data' spans very different practices with distinct privacy/IP risks, and AI companies rarely disclose which they use:
- Pretraining with user tokens as targets: high regurgitation risk.
- Distilling large models into small ones using user prompts: low regurgitation risk, likely common.
- Building RL tasks from user traces: low memorization risk but can extract customer IP, ranging from benign feedback-in-reward-model to invasive setups.
Sarah Hooker adds there are synthetic-data techniques generating distributionally equivalent data while preserving privacy.
More from Safety
- METR self-audit passes most of its own criteria, fails only on conflict-of-interest disclosure — kevinnbass · 2026-09-15
- Big Tech's AI slowdown pact: safety agreement or outright cartel? — The Verge AI · 2026-09-15
- Amodei proposes embedded safety evaluators; Altman and Musk endorse the plan — Miles_Brundage · 2026-09-15
- Kapoor and Narayanan's 13,000-word essay reframes AI loss-of-control incidents — sayashk · 2026-09-15
- Miles Brundage warns of wave of fake DMCA spear phishing attacks on X — Miles_Brundage · 2026-09-15
- New 13,000-word essay: AI safety should bet on control and governance over alignment — sayashk · 2026-09-15