2026-08-04
Princeton trained GPT-2-scale models on zero-free arithmetic; none invented zero at test time, but language pretraining cut the examples needed to learn it by about 48.5% (p=1.7e-4).
The past year has been generous to math-leaning AI. OpenAI and DeepMind report models clearing IMO gold; startups claim 8 of 12 on the Putnam and progress on an Erdős problem. The serious conversation has moved on to what mathematicians do once proofs are automated. This paper (Princeton, with cognitive scientists Thomas Griffiths and Brenden Lake) asks a more basic question that few are asking: can these models reach beyond their training data and posit a genuinely new mathematical object? It picks the smallest possible test case: zero.
Zero did not come easy to humans. The Babylonians used a placeholder to tell 3601 from 61 as early as 1800 BC, but treating zero as a number waited until Brahmagupta's formal definition in 628 AD, more than 2400 years later. Cognitive science has a long-standing hypothesis that language scaffolds this kind of conceptual leap: children use words as placeholders and refine them into formal concepts through experience. The authors put that hypothesis on a neural network.
The design is clean enough to teach from. Train only on zero-free arithmetic; test whether the model can handle arithmetic that contains zero.
The data is single-digit addition and subtraction with two to four operands, e.g. '3+4-2=5'. The split is the whole trick: any problem where zero appears on the left-hand side or is itself the answer goes to the test set, while problems whose answer ends in zero (like '5+5=10') stay in training. Combined with per-digit tokenization ('10' becomes '1','0'), the model does see the '0' token, 2.5% of supervised tokens, but never as a standalone answer or operand. That separates 'has not seen 0' from 'does not understand 0.'
Models are GPT-style decoders, mainly GPT-2 scale (12 layers, 124M parameters), plus an 800K-parameter small model. The dominant confound in any such study is that language pretraining may have leaked in arithmetic. So the authors build a math-free OpenWebText: numbers rewritten as words ('100%' becomes 'one hundred percent'), anything with symbols like + or = removed, leaving 45.1% of documents. Perplexity confirms the model pretrained on this corpus has genuinely not seen arithmetic.
Three experiments: zero-shot (no zero anywhere), few-shot (mix 1 to 1024 zero-examples into training), and 'is zero special' (hold out each digit 1 to 9 in turn, plus a base-8 version).
The cleanest negative result first: no model discovers zero at test time. All four configs (pretrained or not, large or small) have neatly ordered training losses, with the arithmetic-aware pretrained model lowest, but as training loss falls and validation accuracy saturates, accuracy on zero-containing test problems does not move. The appendix's larger Llama-3.2-1B and Pythia-1B (1 billion parameters; given fresh digit tokens to rule out having seen this zero during pretraining) do no better.
Feed in a few zero examples and the picture flips. For the pretrained GPT-2:
| Model | Zero-examples | Zero accuracy |
| Pretrained GPT-2 | 64 (0.64%) | >60% |
| Pretrained GPT-2 | 1024 (10.24%) | >90% |
The headline number concerns sample efficiency. A model with language pretraining needs about 48.5% fewer zero-examples to reach the same accuracy as one without (bootstrap 95% CI 41.2–55.9%, five seeds; t-test p = 1.7e-4). This is the first quantitative neural evidence for the cognitive-science claim that language scaffolds concept discovery.
Zero is special, but not uniquely so; it sits at the hard end of the range. Middle digits (4, 5, 6) generalize best; difficulty rises toward both ends, with zero and the carry digit (9 in base-10, 7 in base-8) hardest. That 7 is hard in base-8 too shows the trouble is the carry operation, not the digit 9. The authors' read: models interpolate but do not extrapolate. Middle digits have more near neighbors in representation space (the count of digits with cosine similarity at least 0.65 forms an inverted V), while 0 and 9 sit at the boundary with few neighbors.
A detail from the error analysis: when zero is held out, the model's most common surrogate answer is −1 (59.13% of right-hand-side errors), followed by 1.
For practitioners, 'small models generalize poorly' is no surprise; the real value here is two things.
First, it draws a clear line under the 'AI is about to invent new math' narrative. IMO gold and Putnam scores are won on problems in-distribution with the training data; genuine discovery requires crossing the distributional boundary to propose new structure. This paper shows that even discovering zero, the smallest conceivable conceptual leap, is beyond current models at test time, and scaling to 1 billion parameters does not help. Benchmark scores and 'can do new math' are separated by a gap nobody has crossed.
Second, the 48.5% number is an engineering signal you can act on: to make a model grasp a concept absent from its training distribution, solid language pretraining roughly halves the labeled examples needed to teach it. The protocol itself (math-free corpus, per-digit tokens, controlled exposure to the 0 token) is a reusable benchmark for conceptual generalization, more meaningful than another MATH leaderboard entry.
The authors flag several: only the GPT-2 architecture was tested, with no reasoning, chain-of-thought, or scratchpad techniques that might be exactly what discovering zero requires; model and data scale are small, and they concede scale alone may explain part of the negative result (the two 1B models are still negative, but larger models remain untested); the error analysis uses a single random seed.
Two caveats stand out. First, the 48.5% saving is measured on a deliberately toy task; whether it transfers to real concept learning, say teaching a model a new scientific concept absent from its corpus, is not addressed at all. Second, the paper shows language pretraining helps but stays silent on how: whether the model transfers a learned notion of 'nothing' or some other mechanism is a black box the authors explicitly leave open. 'Language scaffolds concept discovery' is, for now, a validated correlation with no mechanism behind it.