Incentivizing Self-Learning in Large Language Models via Reinforcement Learning
Lingkai Kong, Yuzhe Gu, Haiteng Zhao, Zijian Wu, Youqing Fang, Zhouqi Hua, Yanan Sun, Kai Chen, Wenwei Zhang
cs.CL
2026-10-29
GRPO plus an exploration bonus and round-level tool credit lifts a 35B model's proof accuracy from 51.5% to 79.5%, 6.4 points above fixed retrieval.
Retrieval-augmented generation (RAG: fetch a document, then answer) and search agents settle whether the passage is in hand. Unfamiliar problems add another loop. A derivation stalls halfway on a missing lemma, and a definition read later rewrites the rest of the plan. Search-style RL interleaves lookup with answering. In-context learning assumes the material is already in the prompt. Meta-RL learns from feedback across repeated attempts. Picking sections from a static long text and writing those principles into a proof had not been trained as its own skill.
The task is textbook proofs. Undergraduate and graduate math books are parsed into chapter-indexed exposition. Exercises and solutions are removed, so the model sees definitions, theorems, explanations, and proofs of general results, and must connect ideas scattered across sections. Problems that cite a section number are dropped. The pool is 5,183 problems from 158 books, 56 sections and about 208k tokens per book on average. The reference solution is hidden from the policy and shown only to the judge.
BookNavigator is the only interface, with three actions. gettoc returns the table of contents. searchindex runs a keyword search scored by term coverage, proximity, exact phrase, and title, and returns section ids plus snippets. readsection returns one full section. Each turn writes reasoning, then takes one action. The observation stays in the history, so the next turn can change direction. The cap is 8 turns and 64k tokens for the whole trajectory. The last turn stops reading and submits a self-contained proof.
Training starts from Intern-S2-Preview-35B and runs GRPO for 50 rollout steps. GRPO, Group Relative Policy Optimization, scores trajectories sampled together by their relative advantage. The training set is 4,557 problems from 132 books. GLM-5.2 grades proofs with 8-shot pessimistic verification: eight independent reviews, reward +1 only if all eight agree the proof is correct, otherwise -1.
Correctness alone teaches a stronger model to open the book less often. Self-learning training adds two terms. Baseline RL does not.
Three held-out splits: ID-Book, 205 problems from 4 books used in training; OOD-Book, 155 problems from 7 unseen books in seen subjects; OOD-Subject, 266 problems from 19 probability and statistics books held out entirely. Accuracy is the mean of 4 sampled proofs per problem.
Fixed baselines cannot call tools while generating, and the output cap is also 64k. No-Book sees the problem only. Top-k prepends at most 8 sections by relevance. Long-Context fills that 64k budget in textbook order. Oracle appends only the section the exercise came from.
| Protocol | Before RL | After RL | Gain |
| No-Book | 55.7% | 68.5% | +12.9 |
| Top-k Sections | 62.2% | 73.1% | +10.9 |
| Long-Context | 56.7% | 72.2% | +15.5 |
| Oracle Section | 60.9% | 71.0% | +10.0 |
| Self-Learning | 51.5% | 79.5% | +28.0 |
Before RL, self-learning is at 51.5%, under No-Book at 55.7% and under every fixed-retrieval setup. After RL it leads at 79.5%, 6.4 points above Top-k at 73.1% and 11.0 points above RL-trained No-Book. On OOD-Book, self-learning reaches 85.2% against Top-k at 81.0%. On OOD-Subject, 86.7% against 81.3%. ID-Book moves from 33.9% to 65.5%, a gain of 31.5 points, the largest of the protocols.
The same weights, with the book and the tools removed, still rise from 55.7% to 71.7%. The paired textbook then adds 7.8 points, to 79.5%. Another book from the same subject scores 75.2%. Deleting the source section plus four sections on either side, nine sections in total, drops accuracy to 74.6%. Deleting nine sections outside that neighborhood leaves 79.8%, next to the intact book at 79.5%.
Outside proofs, each benchmark keeps its own inference setup. BookNavigator is not used. BrowseComp-Plus stays on BM25 keyword retrieval. CL-bench reads the supplied context with no tools. Self-learning RL moves BrowseComp-Plus from 50.2% to 51.9% (+1.7) and CL-bench from 14.8% to 17.1% (+2.3). Domain Knowledge Reasoning goes from 14.5% to 19.0% (+4.5). Rule System Application goes from 16.3% to 17.5% (+1.2). Empirical Discovery ties the No-Book RL run at 12.6%, against a 7.5% base. Procedural Task Execution falls for every RL run; self-learning lands at 15.9% against 16.6%. On BrowseComp-Plus, every other RL protocol finishes below the base model.
Ablations keep the self-learning protocol and change only the reward.
| Setting | Overall | Trajectories with a tool call | Calls per trajectory |
| Base | 51.5% | 36.9% | 1.5 |
| Correctness only | 74.9% | 6.7% | 0.3 |
| Plus tool credit | 76.2% | 7.0% | 0.3 |
| Plus exploration bonus | 77.0% | 79.9% | 5.9 |
| Both | 79.5% | 58.4% | 3.0 |
Correctness-only GRPO already reaches 74.9%, while tool-using trajectories fall from 36.9% to 6.7%. Tool credit lifts ID-Book from 60.7% to 64.0% and leaves tool use at 7.0% of trajectories. OOD-Book sits 0.8 points under correctness-only. The exploration bonus alone pushes tool-using trajectories to 79.9% and 5.9 calls on average, and adds 2.9 points on OOD-Subject, but overall accuracy is 77.0%. Both terms together average 3.0 calls and lead every split.
A book of about 208k tokens does not fit in one window. The policy reads sections while the proof is in progress. The annealed bonus pays only when the proof is right and the book was actually opened. A frozen copy of the initial model shifts credit off calls that did not help.
Correctness-only GRPO already walks the score from 51.5% to 74.9%. The bonus and the round-level credit add the remaining 4.6 points and bring tool-using trajectories back from 6.7% to 58.4%. The cross-book and cross-subject margin lives in those 4.6 points. BrowseComp-Plus and CL-bench gain 1.7 and 2.3 points, and those evals do not use BookNavigator. The recipe holds on proofs. On other long-context tasks the transfer stays thin.
Every accuracy number comes from an LLM judge. GLM-5.2 must agree on all eight votes, which is strict, and there is no human audit and no formal proof check. The reward and the reported scores share that judgment. The tool judge is the frozen pretrained 35B, and useful and neutral share one score, so the process signal only penalizes harmful calls.
The held-out subject is still probability and statistics. Off mathematics the gains shrink to a couple of points, and procedural execution goes backward: proof training pays for building a derivation, while procedural items pay for following operational constraints. Only Intern-S2-Preview-35B is trained, for 50 rollout steps. Retrieval is keyword scoring, with no comparison to embedding search. ID-Book is four books. A trajectory stops at 64k tokens and 8 turns, against books that average 208k tokens.