WISD: weakening a model via quantization or pruning boosts reasoning by up to 5.5 points
songhan_mit · x · 2026-11-03
The WISD paper (Weakening-Induced On-Policy Self-Distillation) improves reasoning without a stronger teacher: deliberately weaken the model via quantization or pruning, then use the original-minus-weak log-probability contrast as the training signal on the model's own trajectories — no external teacher, reference solutions, or reward signals.
Key results:
- Average accuracy gains of +4.0 / +5.5 points with quantization / pruning
- HMMT 2025: 57.7% → 70.5% on Qwen3.5-4B
- LiveCodeBench v6: 39.1% → 56.7% on Qwen3.5-4B
Code is open-sourced on GitHub (Apache-2.0) with a project page.
More from Research
- UMI: Handheld Grippers Turn Human Demos into Hardware-Agnostic Robot Policies — gabriberton · 2026-11-03
- Direct Weight Editing via Codex+Astra Lifts Audio Classifier Accuracy From 84.3% to 88.75% Without Retraining — dansitu · 2026-11-03
- How UMI turns handheld grippers and SLAM into robot training data, in 5 steps — gabriberton · 2026-11-03
- TSFLab: A Self-Improving Agent-Agnostic Workbench for Time Series Forecasting — D2I-Group · 2026-11-03
- YC's Momentic launches QA-Bench: its AI QA agent finds 2.7x more reproducible bugs than Codex — ycombinator · 2026-11-03
- EDU-Jev 4B education decision model hits 67.6%, beating top commercial API by 9.4 points — kalyan_kpl · 2026-11-03