Peking University: post-training leaves behavioral shadows, one word per prompt transfers coding skill
PekingUniversity · hf · 2026-09-29
Peking University introduces Active Taskless Distillation (ATD), showing LLM capabilities can transfer through task-unrelated text.
- Students learn from prompt–word pairs where the teacher outputs just a single word per prompt—no task examples, no logits, no teacher parameters
- On Qwen2.5-1.5B, 5,664 samples yield 5.34 pp gain on HumanEval+ over a nuisance-matched control
- Transfer also appears in scientific knowledge, commonsense reasoning and reading comprehension across generations, sizes and families
- Functional analyses show the learned signal is composable and scales with the teacher's update strength
A sharpened follow-up to subliminal learning research.
More from Research
- ColNanoVDR distills multi-vector visual document retrieval without documents, keeping 95% NDCG@5 at 149M params — nanovdr · 2026-09-29
- NUS rethinks DiT residual connectivity: 1.73x fewer training iterations, 1.39 FID — NationalUniversityofSingapore · 2026-09-29
- Imprint Reader Decodes Weight Updates into Natural Language, Enables Targeted Edits — Guanxu Chen · 2026-09-29
- SJTU's GeoVerse Synthesizes World-Consistent Novel Views in Geometric Latent Space — SJTU · 2026-09-29
- Tencent Hunyuan Maps Scaling Laws for Encoder-Free Multimodal Pretraining — Tencent-Hunyuan · 2026-09-29
- When Do Model Internals Help? Benchmarking Representation Engineering for LLM Safety — Tianyi Guan · 2026-09-29