New paper quantifies ineffable concepts: 65 of 1,270 can never be taught to LLMs via language

sanmikoyejo · x · 2026-10-31

Alexander Spangher and collaborators (including Sanmi Koyejo, Diyi Yang, and Noah Goodman) released Articulation Upper Bounds, presenting a SOTA prompt optimization framework and a novel unsupervised measure of prompt performance.

Core idea: a transmitter LLM writes a prompt conveying a concept to a receiver LLM, which reconstructs the prompt from (items, labels) and relabels them; agreement between the two labeling schemes measures whether the concept was transmitted. This metric outperforms self-consistency and LLM self-judgement for optimizers and closely tracks human labels where ground truth exists.

Scaling findings: fitting observational scaling laws over 1,270 concepts reveals three regimes:

321 concepts are articulable with current models (they reach the threshold), mostly local, self-contained patterns;
884 concepts will become articulable with larger future LLMs (rising), mostly whole-text structural patterns like worldbuilding coherence;
65 concepts are bounded—no LLM size will articulate them, typically those tied to identity and social groups, e.g. "comic voice" or a community's in-jokes.

Implication: the concepts models can't be told are exactly those most wanted for frontier tasks like taste, pluralistic alignment, and discovery, with important consequences for rubric design and learning strategies.

Original post →

New paper quantifies ineffable concepts: 65 of 1,270 can never be taught to LLMs via language

More from AGI Musings