Qwen3.8 27B scores 23.57% on word-based addition without reasoning, 167/169 with it
dl_weekly · x · 2026-10-15
Simon Willison replicated Colin Frasier's GPT-4o experiment on a local DGX Spark using Qwen3.8-27B-Q4KM.gguf, testing 5,070 reasoning-disabled addition cases where results must be written out in English words. Without reasoning, numeric accuracy was just 23.57%, collapsing from 97.04% on 1-3 digit operands to 6.44% on 10-13 digit operands—despite 96.17% format compliance, meaning the model looks right while being wrong. With reasoning enabled, it solved 167 of 169 one-shot cases, with traces showing self-correction mid-calculation. The replication itself was built by pasting a chart into a Codex Remote (GPT-6 Astra) session.
More from Models
- Tester finds GPT-6 Astra still can't solve this robotics problem, predicts it becomes the field's baseline — DJiafei · 2026-10-16
- One prompt got Gemini 4 Argon to build a relativistic particle engine in a single HTML file — vitaliychiley · 2026-10-16
- NVIDIA releases Nemotron 3.5 tech report — eliebakouch · 2026-10-15
- NVIDIA's first Bolt model 74B-A9B scores 21.4 with 39% fewer parameters than its predecessor — ArtificialAnlys · 2026-10-15
- Nemotron 3.5 Super VL hits ~630 tokens/s at 100k-token prompts, fastest 20+ score model measured — ArtificialAnlys · 2026-10-15
- NVIDIA's Nemotron 3.5 Super VL tops US open-weight models at 26 on Intelligence Index — ArtificialAnlys · 2026-10-15