DisaggKD paper splits teacher and student across nodes to lift KD memory limits
Ruixuan Tu · hf · 2026-10-20
A new paper by Ruixuan Tu (University of Tokyo / RIKEN AIP) and Yuyang Dong (SB Intuitions), "DisaggKD: Efficient Knowledge Distillation through Teacher–Student Disaggregation and a Fused Loss":
- Standard KD colocates teacher and student on the same GPUs, so per-GPU memory caps the teacher size
- DisaggKD disaggregates them across nodes: teacher replicas stream hidden states to the student; a KD loss over row windows and vocabulary shards exactly equals the full-vocabulary loss without materializing full-vocab logits; a prefetch ring paces producer and trainer
- Closed-form memory and traffic models locate each layout's memory limits
Results:
- On Qwen3.5-35B-A3B → 9B, the disaggregated layout matches colocated LM loss and runs faster; the fused loss also lets colocated runs use fewer nodes at higher speed
- On larger pairs, colocated runs with the fused loss still OOM while the split trains fine
More from Infra
- IFP report: US AI lead over China is only ~6.5 months, and existing policy could double it — fiiiiiist · 2026-10-20
- NVIDIA's first GB300 FP4 counter-submission still 1.64x worse perf per dollar than TPUv7 — xennygrimmato_ · 2026-10-20
- Long queues at RTX Spark SF launch for local Qwen demos built with ODS software — TheZachMueller · 2026-10-20
- Dream-1 Beta runs robot tasks at $1.72 per success, up to 41x cheaper than LLM agents — gharik · 2026-10-20
- Arm AGI CPU posts 2x end-to-end agentic workflow performance vs leading x86 in Signal65 tests — ryanshrout · 2026-10-20
- Loop engineering is a teeter-totter: you never have enough CPU cores and inference at once — kieranklaassen · 2026-10-20