Small 0.6B post-trained model tanks on MATH due to tiny weight issue, not capability
rasbt · x · 2026-11-04
Sebastian Raschka admits he assumed he was testing a base model, because he didn't expect a post-trained one to be this bad at MATH. Charles Frye clarifies it's the instruct/post-trained variant behaving oddly at 0.6B: the model isn't entirely broken in capability terms but is extremely sensitive to sampling settings and prompting, apparently due to a tiny issue in the weights. Inference-side tweaks can sidestep it — a reliability problem, not a capability one.
Related event: 0.6B post-trained model's poor MATH scores traced to a weight bug(3 posts)→
More from Models
- BioSimBench: frontier LLMs ace biology benchmarks but fail at predicting real cell experiments — alex_verem · 2026-11-05
- Kimi K3.1 Pro gains less from test-time compute than Opus 5.0, data shows — teortaxesTex · 2026-11-05
- Grok and Claude Now Refuse to Root Your Own Devices, Citing Brick Risk — doooyle · 2026-11-05
- Nace AI offers 120% cashback to switch teams to Drex Max, claims 11x cost edge over Claude Code — testingcatalog · 2026-11-05
- Stanford benchmark: 18 AI models tested as teammates, best scores just 82.2/100 — james_y_zou · 2026-11-05
- Kimi K3.1 Pro hits 0.78x human compute multiplier, lagging frontier by ~4.6 months — ChowdhuryNeil · 2026-11-05