rasbt questions Qwen3 weight-editing gains: 50 GRPO steps already lift MATH-500 from ~15% to ~50%
rasbt · x · 2026-11-04
Sebastian Raschka comments on a weight-editing result with Qwen3 and raises a key caveat: the Qwen3 base model, with no fine-tuning and only 50 GRPO steps, already jumps from 15% to 50% eval accuracy on MATH-500.
He suspects the base model can already solve 50% of MATH-500 but is simply poor at following instructions/formats, so the weight editing may mainly improve instruction following rather than injecting actual knowledge. Whether it generalizes to other models and tasks remains to be seen.
More from Models
- Moonshot releases Kimi K3.1-Pro as open weights on Hugging Face — ApprehensiveAd3629 · 2026-11-04
- Kimi K3.1 weights released: Gated MLA layers swapped for Gated DSA for long context — teortaxesTex · 2026-11-04
- Weight-Space Programming: GPT-6 edits just 0.016% of Qwen3.5-0.8B weights, boosting math up to 6x — HaggaiMaron · 2026-11-04
- Moonshot open-weights Kimi K3.1 Pro: 2.8T-param MoE with 1M context — scaling01 · 2026-11-04
- GLM-5.3 goes into production on T-Systems' sovereign Industrial AI Cloud in Munich — jietang · 2026-11-04
- ChatGPT Pro's Dot voice interface feels "scarily human-like," like the movie Her — irinarish · 2026-11-04