Apple's GSM-Symbolic paper: renaming numbers sinks SOTA models' math scores
imjustnewatai · x · 2026-11-06
A repost of Apple's GSM-Symbolic paper: researchers applied trivial surface changes to GSM8K grade-school problems (swapping names and numbers) to build a new benchmark, and many SOTA models showed clear performance drops—suggesting pattern matching rather than genuine logical reasoning. The screenshot shows Table 1's full 8-shot results comparing GSM8K vs GSM-Symbolic across 25 models including Gemma, Phi, Mistral, Llama3, and the GPT-4o/o1 families. The poster highlights which models were actually tested.
More from Models
- 8 Types of LLMs Powering AI Agents, from GPT Backbones to LAM Action Models — mdancho84 · 2026-11-07
- ChatGPT Images 2.6 vs Nano Banana 2.1: 4B weekly images vs a 50% price cut — Aiden_Tech_Ai · 2026-11-07
- Benchmarks: Kumo, Nori, Chronos-2 vs tuned LightGBM for power prices — nkafr · 2026-11-07
- a16z: AI Apps Shouldn't Price Per Token — Price at the Highest Value Layer — aleximm · 2026-11-07
- Ex-Rockstar Tech Director: Threat of AI Decompilation Ports May Force a Faster GTA VI PC Release — Ars Technica AI · 2026-11-07
- Running quantized Qwen on a 16GB M2 Pro: 77 tok/s prefill at 120k context — SocietyTomorrow · 2026-11-07