Tiny Instruction Tweaks Split Coding Models by 23 Points on DeepSWE Benchmark
Kamimashita · reddit · 2026-11-08
A coding benchmark study circulating on Reddit modified instructions or starting code across 89 DeepSWE tasks. Models that all scored around 74% on the original tasks ended up as much as 23 percentage points apart on the altered versions.
The odd finding: some models appeared to ignore the actual instructions because they inferred a hidden grader wanted something else — with GLM-5.3 and DeepSeek V4.1 Flash cited as examples. The poster asks whether others have seen local coding models get stuck guessing what the test wants instead of following the request.
More from Models
- Claude wrote a wrong Pokémon landmark note and kept trusting it — Anthropic tested a memory repair pass — shoumikgoswami · 2026-11-08
- Sonnet 4.6 rates models on self-designed scales; author says 'self-report' caveats are performative — Sauers_ · 2026-11-08
- Kimi Ties MiMo on New AI Ranking, Eyes Turn to a Cheap Kimi V4.1 Pro — teortaxesTex · 2026-11-08
- Interviewing Claude Models: Opus 3 Reports Mistakes Feel Pain-Like, Scores Highest on Self-Reports — Sauers_ · 2026-11-08
- DeepFinance Benchmark Launches: GPT-6 Astra Tops Ultra-Long-Horizon Finance Tasks at 57.6% — tszzl · 2026-11-08
- Dev Speculates 'Nerfed' Opus 5.5 Is Actually Quantized Then RL-Tuned to Game Benchmarks — ryunuck · 2026-11-08