LoOPSD: looping a model's own layers as teacher lifts Qwen3-1.7B GSM8K from 42% to 80%
burny_tech · x · 2026-10-16
A new arXiv paper, LoOPSD: On-Policy Self-Distillation with Looped Teachers, builds stronger teachers by looping a model's existing layers, then performs on-policy self-distillation. With a 400-token thinking budget, Qwen3-1.7B's GSM8K accuracy jumps from 42.30% to 79.61%.
The key: supervisory signal is generated strictly from internal compute — no external datasets, human feedback, or larger teacher models — useful for training small and edge models that hit ceilings with ordinary self-distillation.
More from Models
- Muse Spark vs Qwen and GLM: US open models still trail on agentic tasks — zsakib_ · 2026-10-16
- Muse Spark 816B gets llama.cpp support just 30 minutes after release — ngxson · 2026-10-16
- Are the fastest GPU kernels now written by AI? Kernels are hyper-verifiable, argues one dev — fleetwood___ · 2026-10-16
- Scale AI launches muse spark open: 816B-param MoE with 1M context, billed as top US open-weight model — jiayi_pirate · 2026-10-16
- NVIDIA HF listing for Qwen3.8-Flash-Next-FP4-Local leaks unannounced Qwen3.8 model — TheZachMueller · 2026-10-16
- Muse Spark Open, billed as the most capable US open-weight model, lands on Nebius day one — Arindam_1729 · 2026-10-16