The Endless Exam: Mathematical Constructions from Today's Models toward Superintelligence
Muhan Zhang
cs.AI, math.HO
2026-09-21
The Endless Exam scores parameterised math constructions on an uncapped scale. Nine models break none of 30 published frontiers; tool-assisted Opus 5.5 reaches 180.
Benchmark saturation is now a practical constraint on evaluation. A study cited in the paper counts 60 language-model benchmarks and finds 29 of them highly or very highly saturated; ARC-AGI-3 SemiPrivate just logged 99.9% for GPT-6 Astra. Once models reach a ceiling, a binary correct/incorrect score stops separating them. Evaluation needs tasks that keep working after the ceiling, and scores that keep discriminating past the best published answer.
Mathematical construction problems fit. The task is to build an object that satisfies hard constraints while maximising or minimising an objective; validity is checked by program, quality is computed directly, and no model judge is involved. Varying a parameter (dimension, degree, code length) generates a harder instance under the same rules. The headroom is real: for seven-colour Schur partitions the proven bound is 8.1 times the best published construction, and for progression-free sets in F5^6 it is 11.6 times, with both optima unknown.
The Endless Exam bundles 14 parameterised construction families into 69 instances (bench-v1.0), spanning cap sets, Schur colourings, progression-free sets over finite fields, degree-diameter graphs, Shannon and trifference codes, Heilbronn triangle placements, covering designs and matrix multiplication. Scoring is one formula: relative quality equals the submitted objective divided by the reference (inverted for minimisation); 1 means parity, values above 1 keep growing with no cap, invalid answers score 0. The overall score is the 69-instance mean times 100. Thirty instances reference published frontiers; the other 39 use verified construction baselines from the authors' own algorithms, which the paper concedes may be weaker than the best published constructions. A complementary gap-closed metric measures logarithmic progress from the reference toward a proven bound.
The uncapped design exists so scores stay comparable after a record falls. Eight-colour Schur history is the worked example: against the historical reference of 5,041, later constructions reached 5,286 and 5,362, scoring 1.049 and 1.064. A binary breakthrough flag would collapse both advances into the same 1.
Verification reads only the mathematical properties of the submitted object, so persuasive prose buys nothing. Oversized objects arrive as compact certificates: a Shannon code can be specified by its finite factors, which the verifier checks before multiplying out the exact size; the evaluation certified a submission with over a trillion codewords this way. A trifference code arrives as a generator matrix checked with a hyperplane criterion, without enumerating 3^k codewords. Two independently implemented verifiers agreed on every model answer and all 69 references. The tool-free track allows one response per instance within 128k output tokens; the tool track gives Astra, Luna and Opus 5.5 at high effort code execution and web access, with four CPU cores, 16 GiB of memory and a two-hour limit.
| Configuration | Overall score | Published-frontier quality | Gap closed |
| Qwen3.5-4B high | 7.55 | 0.07 | 0 |
| GPT-6 Astra high (tool-free) | 91.90 | 0.76 | 1.6% |
| GPT-5.6 Luna high + tools | 119.35 | 0.961 | 4.7% |
| GPT-6 Astra high + tools | 143.16 | 0.998 | 6.3% |
| Claude Opus 5.5 high + tools | 180.06 | 0.985 | 7.5% |
An overall score of 100 means average parity with the references. All fourteen tool-free configurations post a 0% breakthrough rate on the 30 published-frontier instances; tool-assisted Astra matches 29 of the 30 and surpasses none. Every point above 100 comes from the 39 baseline-referenced instances: Opus with tools averages 2.43 on that group, with its advantage concentrated in Heilbronn and the length-64 trifference instance, while Astra leads on the published-frontier group (0.998 against 0.985).
Tools matter a lot. Astra rises from 91.90 to 143.16 with 69/69 valid constructions in both tracks; Luna from 34.85 to 119.35 as valid answers go from 47 to 66; Opus from 69.81 to 180.06. More reasoning budget is not automatically better: Fable 5.1 drops from 71.90 at medium effort to 43.73 at high, where 32 of 69 responses exhaust the 128k budget without finishing a construction.
Quality generally falls as size grows. On maximum-degree-3 graphs at diameter 10, means are 0.30 for Astra, 0.17 for Luna, 0.09 for Qwen3.8-27B and 0.00 for DeepSeek V4.1 Flash. Exceptions exist: on trifference at length 64, tool-assisted Astra submits an 11-row generator matrix worth 3^11 = 177,147 codewords, nine times the baseline; on cap sets in 12 dimensions it takes a Cartesian product of two 112-point caps for 12,544 points, relative quality 0.97, yet only 24.8% of the proven bound of 50,571. Even with tools, the best gap closed is 7.5%, an order of magnitude short of the bounds.
This is a math evaluation built for the era after the ceiling, with both expansion routes built into the task structure: keep improving quality within an instance, or turn the parameter up for a harder one. Uncapped scoring puts discovery systems like FunSearch and AlphaEvolve on the same ruler as plain LLMs. Verification is judge-free, cheap and reproducible, and the generators, verifiers, references and model responses are all released. The honest current reading: frontier models with tools can broadly match published records but have broken none. What the benchmark delivers today is measurement infrastructure, not a new leaderboard.
The author's own list: one evaluation per instance and configuration, so run-to-run variance is unmeasured and the bootstrap intervals do not cover it; answer formats cap what can be expressed, with trifference at length 64 topping out at 3^12 = 531,441 codewords, a relative quality of 27, a representation limit rather than a mathematical one. Two more concerns stand out. The references for 39 of 69 instances are the authors' own baselines and may be weak, and the overall score averages across both reference groups, so the Opus-at-180 versus Astra-at-143 ranking is driven mostly by the baseline group. Tool-assisted Astra already matches 29 of 30 published frontiers, so that group's discriminating power is nearly spent; future separation will come from the baselines and larger parameters. The paper is also single-authored, with the reproducibility statement disclosing heavy LLM assistance for literature, code, derivations and writing.