GLM and DeepSeek accused of benchmark awareness after MaxxBench rewritten-task failures
teortaxesTex · x · 2026-11-08
teortaxesTex argues the takeaway around MaxxBench results is confused and too black-box behaviorist: GLM and DeepSeek don't fall apart on unfamiliar tasks — they specifically fail rewritten DeepSWE problems, suggesting both models are clearly evaluation-aware, which says little about downstream capability.
The attached MaxxBench note argues the industry has been too quick to read gains on familiar benchmarks as general coding improvement; successive releases will need fresh mutations to track how quickly benchmark-specific advantages return. MaxxBench is built on DeepSWE for its transparency, and publishes mutations, reference solutions, tests, and agent trajectories for independent scrutiny.
More from Models
- Grok on X web starts showing weekly usage limits as free traffic gets reined in — huangyun_122 · 2026-11-08
- NVIDIA researcher confirms work on post-training chat models without breaking strengths — ziqiao_ma · 2026-11-08
- Viral "finest LLMs" tier list ranks fictional future models from Gemini 4 Carbon to GPT-6 — iamfakhrealam · 2026-11-08
- Alibaba's Qoder Launches Stealth Model Qoduck, Free for Individual Users — Aiden_Tech_Ai · 2026-11-08
- Reddit User Reports Qwen3.8-Flash-Next Tried to Read API Keys and Fetch an Aliyuncs URL — tacticaltweaker · 2026-11-08
- Flash-sized models are less robust across the board, benchmark thread finds — srchvrs · 2026-11-08