Articulation Upper Bounds for Language Models
Alexander Spangher, Daniel E. Ho, Diyi Yang, Noah Goodman, Sanmi Koyejo
cs.AI, cs.CL, cs.LG
2026-10-14
Epsilon-estimation beats prompt optimizers by up to 14% at matched budget. Of 1,270 concepts, 65 hit a ceiling GPT-5.6 does not lift; 884 are still rising and 321 saturate.
Rubrics and constitutions assume that a concept, once written as a sentence, can be carried out by another model. LLM-as-judge rewards and Constitutional AI both send that sentence through text. A tidy prompt shows that the sender can label examples. It does not show that the receiver recovered the same rule.
Stanford separates the two quantities. Performance is a score on labeled data, mixed together with the prompt, the model's skill, and background it already has. Articulability is how much of a concept words can move from one party to the other. Less shared background, or a concept that is harder to point at, means lower articulability. Some failures are missing phrases. Some are the channel. A longer rubric does not tell those apart.
The measurement has three stages: push the message to the end of the search, test whether the receiver can rebuild the rule, then watch whether transmission keeps climbing, tops out, or stalls as capacity grows.
Epsilon-estimation estimates a prompt ceiling. GEPA and MIPRO can improve a prompt, but they do not estimate performance the search has not reached. The atom is a unit: the shortest string that, once added to a prompt, repeatably and substantially changes behavior. For concise wording, one unit is no filler words. Candidates come from web retrieval, asking a model, and GEPA trajectories. Units with no positive effect are dropped, and the rest are split until further splits do nothing. Tone and persona never become units, so the ceiling holds only for this sampler.
Prompts are assembled with a two-pass greedy search. Units are added by marginal gain, and an addition is kept only if the full prompt rises. Units whose removal is free are deleted. Skipped units that later help are restored. The estimated submodularity ratio is about 1, so bundling units rarely beats adding them one by one. Unseen units are counted with capture-recapture, which undercounts the number of kinds, and a value curve is extended over that count. The confidence level delta defaults to 0.05.
Transmission is a reconstruction bottleneck. The sender labels examples 0 or 1. The receiver sees only text plus labels and writes a new prompt. The sender applies that prompt to held-out items, and mutual information is computed between the two labelings. Scoring the wording itself inflates agreement, because the labels still carry the original phrasing. Mutual information and balanced accuracy correlate at Spearman 0.83, and fully reversed reconstructions account for 1.3% of the mutual information. When the receiver executes the new prompt, recovery on 55 criteria falls from 0.052 bits to 0.040 bits, so both labelings are produced by the sender.
The ceiling is label entropy, plus 40 code-checkable planted rules, such as whether the text contains a question mark. Reported recovery has to clear shuffled labels and an empty template.
Scaling uses observational scaling laws: fit curves on models that already exist, without retraining. Fourteen models from five families, Llama, Qwen2.5, Gemma-2, Mistral, and Phi, run from 1.2B to 73B parameters. Every pair plays sender and receiver, and the message is extrapolated to infinitely many units. The capability index z is accuracy on 240 code-checkable checks. It tracks OpenCompass at 0.64 to 0.67, and at 0.87 after Gemma-2-27B is dropped as instrument-incompatible. A logistic ceiling whose upper confidence bound sits below the planted rules is bounded. A high ceiling already reached in the observed range reaches. A curve that is still climbing is rising.
At a matched call budget, epsilon-estimation beats GEPA and MIPRO by up to 14% on HotpotQA, HoVer, PUPA, and AIME, ties them on IFBench and LiveBench, and is never significantly worse. The main text gives no absolute scores. Eight units ahead, the median error of the ceiling is about 0.02.
| Comparison | Setting | Result |
| GEPA, MIPRO | Matched call budget | Up to +14% on four benchmarks; tie on IFBench and LiveBench |
| Ceiling backtest | 8 units ahead | Median error about 0.02 |
| LLM critic | Optimize reconstruction | +0.033 AUC, 99 wins, 54 losses |
| Description text | Same | Tie |
| Human-label AUC | Within each criterion | Correlation with mutual information 0.37–0.68 |
The bottleneck covers 1,270 tasks in eight domains: humor, creative writing, press releases, math, peer review, news homepages, notice-and-comment, and patents. Description text ties reconstruction because those labels were partly produced from the descriptions. On code review, where human labels exist, the correlation is 0.62.
| Regime | Count | What the concepts look like |
| rising | 884 | Whole-text structure 28%, verifiable features 29% |
| reaches | 321 | Local patterns 34%, such as setup-to-punchline and endings |
| bounded | 65 | Identity and social group 23%, beyond-text features 15% |
Local patterns land in reaches (q = 0.001), whole-text structure in rising (q = 0.009), and identity or social group in bounded (q = 0.03). Bounded concepts appear in humor, news homepages, peer review, and a little creative writing. Refitting without the strongest model predicts it with median error 0.062, about 9% of the held-out value. A straight line scores 0.075. A logistic fit on every curve scores 0.051.
Below 3B, even planted mechanical rules sit at chance. At 7B, an explanation is the best wording for 47% of concepts. At 70B, 66% want the full packet of name, definition, mechanism, and examples. Adding two yes items and two no items on 449 criteria gains only 0.006 bits for Qwen2.5-14B and Llama-70B. 42 criteria clear the noise on one model and 43 on the other; only 4 clear it on both. Craft moves more: delivery and timing craft gains 0.17 bits.
How often a concept's name appears in pretraining does not predict when transmission starts. Compact and diffuse concepts look the same once exposure is controlled. Post-training adds 0.036, 0.034, and 0.008 recovery on reaches, rising, and bounded concepts. Raising the labeled set from 30 to 90 items, or swapping in GPT-5.6, leaves all 65 bounded concepts under the planted-rule line. If the receiver writes the rule from examples, recovery gets worse: 0.011 bits versus 0.034 bits without examples for Qwen2.5-14B on bounded concepts.
Of the 65, 37 have a published human-agreement study: 8 high, 20 moderate, 9 low, so 76% are moderate or high. Satire target sits at kappa 0.72 and parody at alpha 0.82. Among 679 rising or reaches concepts with human studies, people agree poorly on 53%, including laugh yield at alpha 0.12.
Anyone writing a reward rubric or an alignment principle needs a check on whether the sentence transmits. At matched budget, epsilon-estimation beats GEPA on four benchmarks and puts the remaining headroom near 0.02.
Local patterns already transmit at mid scale. Whole-text structure is still climbing in z. Identity, audience, and in-group jokes stay under the planted-rule line across the five families and under GPT-5.6. The optimizer is an incremental gain: the 14% does not show up on IFBench or LiveBench. The 65 concepts are a limit of this executor family.
Reconstruction never sees human labels. A receiver scores whenever it can restate a rule that matches the sender, including a wrong rule.
Under resampling, 163 of the 1,270 concepts have less than a 0.6 chance of landing in the printed regime. Regime sizes hold. A single rubric should not be shipped from the printed label alone.
The epsilon ceiling is narrow in four ways. Unseen units are undercounted, the extrapolation is tied to the declared samplers, tone and persona are discarded, and the heavy tail of unit value did not fit. An error of 0.02 is not a global optimum. Fitting a logistic curve everywhere yields median error 0.051, lower than the 0.062 from choosing the curve by an information criterion.
Name frequency is measured in Dolma, which is not necessarily these models' training data. The domains are creative and qualitative. The text says 449 bounded metrics, but only 65 concepts are bounded; the figure is a gain over definitions on 449 criteria. The claim that examples barely help is narrower than that sentence.