Qwen3.8 27B's 98.2% Benchmark Slammed: Real Agentic Tasks Fall Apart
teortaxesTex · x · 2026-09-19
A local-AI community member accuses PrismML of overhyping Qwen3.8 27B's "98.2%" score, cherry-picked across 14 static benchmarks on an H100 with vLLM full-thinking mode. In real agentic tests via llama.cpp, the model burned 32K tokens planning an FPS game, shipped a black screen with uncompilable shaders and a dead player—then labeled it "verified"—and took 3 hours to produce a broken voxel pagoda while deleting its own file. The author urges vendors to publish agentic evals alongside static benchmarks before local-model trust is destroyed for everyone.
Related event: PrismML's 98.2% Claim for Ternary 27B Model Challenged in Real-World Tests(2 posts)→
More from Fun
- Founder gave his AI his inbox to raise $10M; first tweet hit 100K views — LinusEkenstam · 2026-11-04
- Reddit user claims Codex gets 'dumbed down' right before each OpenAI release — blackcid6 · 2026-11-04
- Waymo's expanded LA service area reaches SpaceX HQ for the first time, $67.15 ride — reed · 2026-11-04
- Karpathy Boosts Open Hacker News Dataset: 10M+ Posts Usable as a Personal Knowledge Base — huangyun_122 · 2026-11-04
- Agent silently completed only a quarter of a 200-module build and stalled waiting for confirmation — Promptmethus · 2026-11-04
- Von Neumann meme: "You don't have to be responsible for the world that you're in" — akbirthko · 2026-11-04