Local Qwen 3.8 Benchmarks: 3-9% Failures Due to Infinite Reasoning Loops
on_line187 · reddit · 2026-08-21
The author ran local benchmarks for Qwen 3.8 27B Instruct (Q80 GGUF) on dual RTX 3090s across GSM8K, MATH-500, HumanEval, and MBPP using mechanical grading (unit tests/sympy, no LLM judges).
Key Findings:
- Overall Scores: GSM8K 96.7%, MATH-500 86.4%, HumanEval 95.5%, MBPP 80.0%.
- Infinite Loop Defect: 3-9% of items triggered a non-terminating reasoning state, consuming the token budget with no output. Excluding these stalls raises MATH-500 score from 86.4% to 94%. This defect also wasted significant compute time.
- Difficulty Decline: MATH-500 scores showed a monotonic decline by difficulty level, with Geometry being the weakest subject (78.0%).
- Contamination Test: Using a continuation-similarity metric, HumanEval showed significant signs of memorization (0.387 similarity vs 0.080 baseline), while MBPP appeared relatively clean.
More from Models
- Muse Spark Open runs locally on a Mac M3 Ultra via llama.cpp: frontier model on desktop — pcuenq · 2026-10-16
- OpenAI ships GPT-Live-1.1: new voices in six languages, fewer confirmations — OpenAI · 2026-10-16
- Muse Spark model appears on LMArena, testable in Battle and Agent modes — arena · 2026-10-16
- Running a multi-model UI design arena: Paseo orchestrates four design drafts per model — vista8 · 2026-10-16
- Ilya Sutskever questions Meta license clause taxing MaaS providers above $20M revenue — ilyasu · 2026-10-16
- Muse-Spark open-weights drop: 816B-A42B MoE model gets 359GB MLX 3-bit build — ns123abc · 2026-10-16