0.6B post-trained model fails MATH: tiny weight defect makes it sampling-sensitive
lateinteraction · x · 2026-11-04
Replying to Sebastian Raschka, Charles Frye explains that the 0.6B instruct model's poor MATH performance isn't a capability issue but a reliability one: the model is unusually sensitive to sampling settings and prompts, apparently stemming from a tiny issue in the weights, and the failure can be sidestepped via inference tweaks.
Related event: 0.6B post-trained model's poor MATH scores traced to a weight bug(3 posts)→
More from Models
- Kimi K3's compute load too heavy to serve economically, analysis argues — teortaxesTex · 2026-11-04
- Kimi K3.1 Pro leaked doc shows Gated DeepSeek Sparse Attention for long context — zephyr_z9 · 2026-11-04
- Nathan Lambert: Chinese open-model labs have never missed a weights release promise — natolambert · 2026-11-04
- MIT: GPT-6 Astra rewrites other models' weights directly, lifting Qwen math accuracy from 7.9% to 48.1% — The Decoder · 2026-11-04
- Moonshot releases Kimi K3.1-Pro as open weights on Hugging Face — ApprehensiveAd3629 · 2026-11-04
- Moonshot open-weights Kimi K3.1 Pro: 2.8T-param MoE with 1M context — scaling01 · 2026-11-04