LLMs Fall for Common Sense Traps: Salience Bias Causes Reasoning Failures
rohanpaul_ai · x · 2026-08-03
The paper Would You Walk to the Car Wash? reveals the salience bias in LLMs during commonsense reasoning: when prompts contain explicit numbers or procedures, models often ignore unstated physical prerequisites (e.g., a car must be driven to a car wash, not walked).
- Benchmark: Introduces SaliTrap, featuring 1,145 prompts across 4 trap types, tested on 12 models.
- Poor Performance: The best model avoided the trap in only 54.8% of queries, with 8 out of 12 models scoring below 30%. More numerical distractors led models to calculate blindly rather than spot contradictions.
- Awareness ≠ Avoidance: Even when models recognized the trap (like GLM-5.1 and Kimi-K2), they still complied with the flawed premise 86.2% and 81.8% of the time.
- Evaluation Advice: Agent evaluations should explicitly test premise-checking as a behavioral control rather than assuming it from general reasoning scores.
Related event: ICML Paper Reveals LLMs' Salience Bias in Common Sense Reasoning(3 posts)→
More from Models
- GYRO quant of Qwen3.8-Flash-Next tested: near Q3 quality at much smaller size — PlasticRevenue4601 · 2026-10-16
- Open source is so back: dozens of open models dropped within 48 hours — beffjezos · 2026-10-16
- Gemini users hit usage caps on 3.1 Pro and 4 Argon, resets not until Oct 16 — R3d-Gr33n-Blu3 · 2026-10-16
- Gemini desktop app can't accept images — an image model that can't take images — Rebl · 2026-10-16
- Apollo Research: GLM-5.3 unprompted reasons about its actions becoming training data — _lewtun · 2026-10-16
- Opus 5.5 turns heads with surprisingly polished vertical video demo — FinanceYF5 · 2026-10-16