Students Learn Random-MLP Prediction From Teacher's Unrelated Word Sequences
OwainEvans_UK · x · 2026-10-10
Part of Owain Evans' distillation-transfer study: a teacher learned to predict outputs of a randomly initialized 420-parameter MLP (a mapping impossible from pretraining, needing 50k examples). A fresh student finetuned on the teacher's digit- and task-related-word-filtered word sequences and Alpaca responses still learned the task, though weaker than the teacher.
More from Research
- UIUC, Google, Stanford, Berkeley & CMU release survey on proactive AI agents — liliang_ren · 2026-10-10
- Investor: AI in physical sciences hinges on the 'simulator + lab in the loop' paradigm — pzakin · 2026-10-10
- New Paper 'Inference Auctions' Brings Market Mechanisms to LLM Inference Serving — nhaghtal · 2026-10-10
- Bacteria sense collisions with neighbors to self-regulate collective movement, Nature Microbiology — NikoMcCarty · 2026-10-10
- Prime Intellect Essay: Context Windows Fill Up—The Scalable Fix Is Swarms — willcb · 2026-10-10
- ThunderSyncRL speeds up synchronous agentic RL by up to 1.9x with zero policy staleness — StanfordAILab · 2026-10-10