Qwen base model looped on 93% of MATH-500, casting doubt on claimed 6x gain
paul_cal · x · 2026-11-03
A claimed 6x improvement (attributed to "Astra") looks inflated: the Qwen base model was stuck in loops, with 93% of its MATH-500 answers running to the 32k token limit — a behavior Qwen's model card warns about. Deleting a single direction behind a behavior is a known trick (previously used to strip refusals), and Astra needed 50 rounds of model runs to find it. On Gemma 270M, which doesn't loop, the change added nothing — so the 6x was measured against a broken baseline.
More from Models
- AgentTime v1.1: GPT-6 Astra tops agent time-awareness benchmark at 51%, Claude trails — maksym_andr · 2026-11-03
- DeepSeek V4.1 Flash 1017 doubles ARC-AGI-3 score to 5.81%, leads open models — teortaxesTex · 2026-11-03
- Trajectory launches post-training platform to let anyone train, eval and deploy trillion-parameter models — yenkel · 2026-11-03
- Sol 6.1 still best cost-per-intelligence worker; Fable 5.5 more well-rounded — haider1 · 2026-11-03
- Anthropic halves its Max 20x plan usage and renames it "Ultra" — EntityRemovalGuy · 2026-11-03
- Critics tear apart "Weight-Space Programming": Atari test spoon-feeds answers, Qwen case fails on Gemma — anshulkundaje · 2026-11-03