Scaling Mathematical Proof Supervision from Structured Textbooks
Youqing Fang, Zhouqi Hua, Yinhao Tang, Yuzhe Gu, Lingkai Kong, Bin Liu, Weiming Zhang, Kai Chen, Yanan Sun, Wenwei Zhang
cs.CL, cs.AI
2026-10-30
Fine-tuning Qwen3-30B-A3B on 380K textbook proofs from 2,118 books raises a six-benchmark mean from 29.17% to 45.83%, ahead of three public math SFT sets.
Math supervision usually arrives as one question glued to one solution. Contest archives, web exercises, and synthetic rewrites all collapse into that shape. A textbook is arranged as a course: definitions introduce a notion, lemmas and theorems build along prerequisites, and chapter order is the prerequisite order. Earlier textbook datasets mostly mined raw text or exercises. The table of contents, cross-references, and steps a proof skips with the word obviously never became supervision you could scale. An isolated QA pair is a flash card with the syllabus torn off.
TOME-Math keeps the syllabus. Researchers at USTC, Shanghai AI Laboratory, Fudan University, and Shanghai Jiao Tong University turn 2,118 university textbooks across 29 subjects into 380,292 proof QA pairs, with chapter order, proposition-proof links, dependency references, and source locations left intact.
DeepSeek-OCR converts each PDF to Markdown. gpt-oss-120b rebuilds the table of contents, and a state machine labels every page as the current chapter, the next chapter, or a boundary inside the page. Exposition is split from exercises. Exercises are evaluation-only, so the model cannot memorize printed solutions.
Proof QA pairs a theorem, lemma, or definition with the proof printed in the book and links the earlier propositions that proof uses: 205,861 items, 88.9% of them extracted by gemini-3-flash-preview. Gap QA targets the jumps. A model marks a missing obligation, restates it as a problem that can be answered on its own, and writes the missing proof. Of 918,235 candidates, 174,431 remain, a 19.00% yield, and gpt-5-mini writes 53.7% of that stream. A judge filters on five axes including accuracy. Invented steps fail. Mixing a thinking model with a cheaper lightweight model costs about $7 per book.
The saved fields are then experimental knobs. At inference the weights stay frozen and every retrieval policy gets the same context budget: closed book, random passages from the same book, the local chapter plus the one before it, dependency-guided propositions, and those propositions plus Gap QA. In training the same questions are ordered four ways: shuffled, forward through the book with gaps first, reversed, or forward with gaps mixed into the chapter. Forward versus reverse tests chapter direction. Gaps-first versus mixed tests whether the skipped steps should be studied before the full proof.
Full fine-tuning starts from Qwen3-30B-A3B-Base, which the paper treats as a 30B base, for one epoch. Sixty-four books are held out entirely, removing 15,661 records and leaving 364,631 training examples. The main test is 300 problems from those books, 50 from each of six subjects. 227 are proofs, and a mathematics PhD accepted every item. Sixteen samples are drawn per problem. gpt-5.6-luna reviews each sample eight times, and one substantive error rejects it. The suite also includes 145 short objective items plus MiniF2F, ProofNet, MATH-500, and AIME 2026.
Same base model, same fine-tuning setup, different corpus. The unweighted mean of six scores moves from 29.17 to 45.83, a gain of 16.7 points. TOME-Math is first on five. On AIME 2026 it scores 10.19, behind MathInstruct at 11.98.
| Corpus | Long proof | Objective | MiniF2F | ProofNet | MATH-500 | AIME 2026 | Avg |
| Base | 13.25 | 32.93 | 34.45 | 17.14 | 68.80 | 8.44 | 29.17 |
| OpenMath1 (1.82M) | 20.16 | 33.86 | 22.87 | 19.25 | 70.40 | 8.21 | 29.13 |
| MetaMath (395k) | 28.73 | 34.66 | 40.70 | 22.55 | 71.20 | 9.98 | 34.64 |
| MathInstruct (262k) | 35.87 | 40.69 | 52.05 | 32.93 | 73.05 | 11.98 | 41.10 |
| TOME-Math (364.6k) | 43.86 | 43.96 | 57.43 | 42.98 | 76.55 | 10.19 | 45.83 |
Size does not explain the ranking. OpenMath1 has about 4.8 times as many examples and still loses all six, including a MiniF2F drop from 34.45 to 22.87. The paper says the corpora differ in more than structure, so this table does not by itself show that textbook organization is the cause.
A retrieval study on the frozen base model separates that factor. With the context budget matched, the single-sample pass rate on the 300 long proofs rises from 13.25 closed-book to 17.38 with propositions plus Gap QA. Random same-book context scores 14.00, the local chapter 14.71, and propositions alone 15.54. Gap QA adds 1.84 on top of propositions. All six subjects rise. Analysis and topology gain 6.00 each. Same-book text chosen without the dependency index sometimes does nothing or hurts.
Curriculum order was tested in 18 cells: six subjects times token budgets of 0.75M, 1.8M, and 2.4M. Forward order with gaps first wins every cell. The shuffle schedule's pass@16, meaning one accepted solution among 16 draws, is 41.04, 42.52, and 43.21. Forward-with-gaps-first reaches 45.38, 48.98, and 50.93. Its lead over shuffling grows from 4.34 to 7.72 points. Against the reversed schedule the leads are 5.65, 7.41, and 8.50. Teaching gaps before the full proof, with direction held fixed, adds only 2.37, 1.91, and 1.53. Chapter direction is the larger effect, and it grows with scale.
At a fixed 0.75M tokens, spreading the data from 8 books to 16 raises that pass rate by 1.6 to 4.8 points in every subject, 3.3 on average. A checkpoint trained on one subject scores 0.92, macro-averaged, of the matching specialist on 50 problems from each other field. Topology and differential geometry reach 1.29 on probability. In the full-corpus ablation, dropping Gap QA costs 5.05 to 6.77 points on four benchmarks, the largest cut. Mixing gaps into the chapter costs 3.74 and 2.59 on the two TOME tests, and shuffling on top of that costs another 0.25 and 0.95. On the full corpus, gap-first order moves the score more than chapter direction, the opposite emphasis from the per-subject study. The paper does not reconcile the two.
With questions and tokens locked, reading the book front to back still beats a shuffle and still beats reading it backwards. Writing the skipped steps as their own problems, and covering twice as many books on the same budget, each add a further gain.
On long proofs and ProofNet, about 364k textbook items beat about 1.82 million OpenMath1 items. MATH-500 is 76.55 against 73.05. AIME is where MathInstruct still wins. Scaling proof supervision and scaling contest drill are different jobs. The authors say they will release the corpus. It is a fit for university proof fine-tuning and for experiments on training order.
The studies that isolate structure use only six subjects. The ten largest subjects hold 84.4% of the records. Mathematical analysis contributes 66,409 items from 307 books. Graph theory contributes 24,406 from 25 books. Whether the curriculum effect survives that imbalance is untested.
Automatic checks that the record is grounded in the source pass on under 50% of items, with a 28.6% fail rate. A human-review flag for missing verbatim evidence covers 48.4%. Gap proofs are written by a model, not copied from the page. In competition mathematics the top quality tier is 21.6% of records and the bottom tier is 71.5%. The paper never states what fraction of the filled-in proofs are actually valid.
The headline numbers are model judgments. gpt-5.6-luna can reject a solution with one vote out of eight. An appendix has mathematics PhDs blind-label 600 responses to estimate false accepts and over-rejection. MiniF2F and ProofNet are scored as natural-language proofs, so a pass is not a machine-checked formal proof. One base model and one epoch do the full-corpus work. OpenMath1 sees far more tokens in that epoch, so the smaller corpus wins without a matched compute comparison.