Qwen 3.8 27B beats Muse 30B on benchmarks but flubs multi-question prompts, user finds
octagoncat23 · reddit · 2026-09-17
A Reddit user reports growing distrust of benchmarks: Qwen 3.8 27B outscores Muse 30B, but in his testing Muse is exponentially better at long-context adherence and multi-step reasoning.
- Notable failure mode: when asked multiple questions in one prompt, Qwen answers only one of them about 50% of the time, spending most of its thinking on a single issue—possibly a side effect of benchmarkmaxxing.
- The user asks the community which models are sleepers overlooked due to benchmark hype.
More from Models
- Open-source VoxLock tops Krisp voice isolation benchmark with 4.95% WER — Arbaz Siddiqui · 2026-11-02
- Anthropic's 85% inference margin sparks debate on open source's price-pressuring role — code_star · 2026-11-02
- User reports xAI appears to have quietly raised weekly usage limits, dropping his meter from 93% to 47% — XFreeze · 2026-11-02
- Meituan's LongCat-2.5 lands as 1.6T MoE agent model, with DeepSeek copycat speculation — teortaxesTex · 2026-11-02
- Asked what AI firms want you to stop owning, Claude said: your own compute — alex_verem · 2026-11-02
- Is Hermes Still Worth It? Codex's $200 Tier Now Overlaps Most of Its Value — Foreign_Lead_3582 · 2026-11-02