CaRE-KD accepted at NeurIPS 2026: letting students decide when to trust distillation teachers
Tanmoy_Chak · x · 2026-10-03
- In knowledge distillation, teachers hallucinate, yet standard KD forces the student to copy them anyway, wasting probability mass on noisy low-confidence predictions and overwriting correct student priors.
- CaRE-KD (accepted at NeurIPS 2026) lets the student decide when to trust the teacher:
- Token level: a confidence gate switches between mean-seeking and mode-seeking divergence based on teacher confidence.
- Sequence level: a rejection mechanism drops updates where the teacher is epistemically less reliable than the student.
- Preprint available.
More from Research
- Loopback Transformer matches 3B model with 75% of parameters, gains mostly from decay — pbaylies · 2026-11-23
- RoboQuest benchmark: best AI agent solves only 23.2% of embodied exploration tasks — omarsar0 · 2026-11-23
- OpenAI publishes internal pretraining perplexity curve, proposes perplexity as shared safety bar — merettm · 2026-11-23
- AI math swarm proves a 962-variable quartic is minimal; 4,973-variable version spans 1.62 MB of text — gleech · 2026-11-23
- Even OpenAI's quasi-Riemann result already spawned vast expository work, argues AI-proof skeptic — gleech · 2026-11-23
- CMU's ExploitBench ranks LLM cyber agents on a capability ladder, not pass/fail — goyal__pramod · 2026-11-23