8B model's 81.6% DeepSWE score is misleading — it's verifier, not generator, says researcher
yifeiwang77 · x · 2026-09-24
yifeiwang77 clarifies that an 8B model's reported scores of 81.6% on DeepSWE and 87.6% on Terminal-Bench 2.1 are misleading: the model is used as a verifier/classifier of answers produced by other models (e.g., fable), not as the answer generator. He argues it's hard to claim this as '81.6% on DeepSWE'. The confusion stems from a earlier tweet questioning how an 8B model could achieve such numbers without anyone being surprised.
Related event: 8B Model's 81.6% Coding Score Questioned: It's a Verifier, Not a Solver(2 posts)→
More from Models
- Do typos discomfort LLMs? Tokenization quirk sparks debate — menhguin · 2026-11-11
- Alibaba releases Qwen4-Max preview with stronger agentic coding and 3D modeling — vista8 · 2026-11-11
- MiniMax H3 hits 100 days open-source: 1,472 derivative models, 148M+ downloads — linoy_tsaban · 2026-11-11
- Meituan's LongCat-2.5 trends on Hugging Face as MoE multimodal model — meituan-longcat · 2026-11-11
- Fable 5.2 hands-on: great at well-defined hard problems, best since GPT 6 Astra — rachittshah · 2026-11-11
- The next AI vendor lock-in won't be the model API, it'll be the harness — Arindam_1729 · 2026-11-11