Counterfactual benchmark shows 11 frontier models fail to grasp programming semantics

jessyjli · x · 2026-11-17

A CosmicAI team (Aditya Thimmaiah, Jiyang Zhang, Jayanth Srinivasa, Junyi Jessy Li, Milos Gligoric) presented work at ICML and IMPL showing that even frontier models lack truly generalizable understanding of programming language semantics.

Core question: can LLMs condition reasoning on explicit formal rules when those rules conflict with training priors? The results show heavy reliance on priors over the given rules.

Original post →

More from Models

Models channel →