Counterfactual benchmark shows 11 frontier models fail to grasp programming semantics
jessyjli · x · 2026-11-17
A CosmicAI team (Aditya Thimmaiah, Jiyang Zhang, Jayanth Srinivasa, Junyi Jessy Li, Milos Gligoric) presented work at ICML and IMPL showing that even frontier models lack truly generalizable understanding of programming language semantics.
- Built the first counterfactual dataset for LLM reasoning on PL semantics: one variant swaps the meanings of familiar symbols like "+" and "-"; another strips syntactic familiarity entirely by mapping addition/subtraction semantics to characters from the rarely-seen Caucasian-Albanian script.
- Evaluated 11 frontier models (reasoning and non-reasoning) on solving program-execution tasks conditioned on these explicitly supplied counterfactual rules.
Core question: can LLMs condition reasoning on explicit formal rules when those rules conflict with training priors? The results show heavy reliance on priors over the given rules.
More from Models
- AA-PersonalityBench ranks Gemini 4 Argon most metaphorical model, GPT-3.5 Turbo least — ArtificialAnlys · 2026-11-17
- Harness Adapters: RL-trained LoRA lifts Qwen3.8-27B on Pi from 29% to 63% on SWE-bench Pro — LysandreJik · 2026-11-17
- Qwen says offensive-cyber training data was removed — placebo or real lobotomy? — Rare_Subject2343 · 2026-11-17
- 12 open speech models tested on 208 drug names: even the best gets 27 wrong — MaziyarPanahi · 2026-11-17
- d1-Omni proves a strict VLM judge, rejecting bboxes accepted by Gemma and LFM2.5-VL — mervenoyann · 2026-11-17
- KORMo-Web-11B, a Korean VLM for GUI and web agents, trends on Hugging Face — KORMo-VL · 2026-11-17