LLMs Fall for Common Sense Traps: Salience Bias Causes Reasoning Failures

rohanpaul_ai · x · 2026-08-03

The paper Would You Walk to the Car Wash? reveals the salience bias in LLMs during commonsense reasoning: when prompts contain explicit numbers or procedures, models often ignore unstated physical prerequisites (e.g., a car must be driven to a car wash, not walked).

Benchmark: Introduces SaliTrap, featuring 1,145 prompts across 4 trap types, tested on 12 models.
Poor Performance: The best model avoided the trap in only 54.8% of queries, with 8 out of 12 models scoring below 30%. More numerical distractors led models to calculate blindly rather than spot contradictions.
Awareness ≠ Avoidance: Even when models recognized the trap (like GLM-5.1 and Kimi-K2), they still complied with the flawed premise 86.2% and 81.8% of the time.
Evaluation Advice: Agent evaluations should explicitly test premise-checking as a behavioral control rather than assuming it from general reasoning scores.

Original post →

LLMs Fall for Common Sense Traps: Salience Bias Causes Reasoning Failures

More from Models