Microsoft Research: Subtracting Weak Models' Logits Helps 8B Student Beat Experts
theomitsa · x · 2026-08-02
A new research paper from Microsoft proposes a novel approach to post-training and distillation, addressing the bottleneck where frontier student models lack a larger teacher to learn from.
Instead of relying on a massive teacher model, the method takes two smaller, weaker models (e.g., a 4B RL model and its pre-RL base), subtracts their logits to isolate the exact "capability direction" that needs boosting.
By feeding this signal to an 8B student model, it ultimately outperforms the domain expert models guiding it in math and coding tasks. This technique unlocks latent capabilities efficiently without the massive compute typically required for a giant teacher.
More from Research
- Simplex Diffusion Models paper talk by EPFL/DeepMind's Justin Deschenaux on Oct 19 — ArnaudDoucet1 · 2026-10-16
- Meituan LongCat team's Data World synthesizes data environments to train data agents — meituan-longcat · 2026-10-16
- FDE-Gym benchmark shows huge gap on real AI delivery tasks: top model scores 0.563 — DeepLethe · 2026-10-16
- ForcingBooster halves computation of autoregressive video diffusion with patch-wise denoising — VAST-AI · 2026-10-16
- Baidu open-sources FlashMask: up to 1.55x faster than FlashAttention-4 for long-context training — BaiduResearch · 2026-10-16
- SemEval-2027 Task 9 opens: multimodal news framing analysis across 4 languages — preslav_nakov · 2026-10-16