LittleLearner: Language Models Under Pedagogically Controlled Knowledge Exposure
Fanfei Li, Jana Zeller, Manuel Prada-Corral, Thaddäus Wiedemer, Prasanna Mayilvahanan, Ryan Cotterell, Wieland Brendel
cs.CL, cs.AI, cs.LG
2026-08-14
A 5B model pretrained on 88B tokens of K-5-only text has a sharp, verifiable knowledge boundary; scaling, GRPO, and ICL all act strictly inside it.
The main obstacle to studying how language models acquire knowledge is that nobody knows what they saw before training. Data contamination inflates benchmark scores, brittle generalization keeps surfacing, and it stays impossible to separate genuine new capability from eliciting knowledge already in the web corpus. Prior work attacks this at evaluation time with harder, leak-resistant benchmarks. Novelty has to be re-audited for every release, and the training distribution itself remains opaque.
This paper flips the direction: constrain the training distribution instead of the eval. A developmental boundary built on the US kindergarten-to-grade-5 curriculum caps concepts, vocabulary, and reasoning demands at elementary level, and a 5B model is trained from scratch inside it. What the model knows and does not know is specified in advance for the first time.
The corpus, LITTLECURRICULUM, is an 88B-token subset of FineWeb-Edu produced by a five-stage pipeline. An Age-of-Acquisition pre-filter drops documents where more than 5% of words exceed a 12-year-old level (AoA is a psycholinguistic table of when words are learned; 10% of words are uncovered and get imputed from Zipf word frequency). Gemini Flash then annotates a subset against Common Core State Standards, with prompts auto-optimized via DSPy and OpenEvolve; labeling all of FineWeb-Edu was priced at roughly USD 46M, so the annotations train downstream classifiers instead. FastText does a coarse pass, and a ModernBERT classifier 50x more expensive runs on the 266M documents where it agrees with AoA. Regex filtering removes algebraic notation (quadratics, Σ, ∂), cutting only 0.1% more. A final frequency-sampling stage drops documents dense in beyond-K-5 terms.
On CommonCoreText, the pipeline retains near-zero beyond-K-5 documents at the cost of keeping only 35-42% of true K-5 text. That is deliberate: 65% of real classroom text is discarded so the boundary stays clean enough for causal readings.
LittleLearner uses the Qwen3-dense architecture, pretrained 100 hours on 8 B200 GPUs with Muon and MXFP8 compute. The cooloff mixture is 91% K-5 pretraining data, 5% K-5-rewritten mathematical reasoning from MegaMath-Pro, and 2% + 2% instruction data, all rewritten to K-5 level by Gemini Flash. Even the tokenizer is trained on the K-5 corpus alone, to keep beyond-scope content from leaking through the vocabulary. Controls: Unfiltered (same recipe, same budget, no filtering) and Gemma 2B (2T tokens) as an external reference.
The boundary is real. On CLEAR, LittleLearner's bits-per-byte climbs with text difficulty while Unfiltered and Gemma 2B stay flat. On Jeopardy science questions split by curriculum scope, the two models are comparable inside K-5, and LittleLearner collapses beyond it while Unfiltered holds steady on both sides.
Behavior matches the setup. Asked what gravity is, it answers "the force that pulls everything down." Asked about Schrödinger's cat, it answers "a cat with two faces." There is no quantum mechanics in the corpus, and the model genuinely has none.
Three knowledge-injection experiments agree:
| Intervention | Inside K-5 | Beyond K-5 |
| Scale 0.6B to 5B | large gains | partial at grades 6-7, floor at grade 8 for all sizes |
| SFT + GRPO | both models improve | LittleLearner barely moves, Unfiltered gains a lot |
| 3-shot CoT | modest | zero gain; explanations also do nothing |
Two details deserve attention. The 0.6B LittleLearner beats Unfiltered inside K-5: with no high-school algebra competing for capacity, the small model is more focused. And post-training LittleLearner on K-5 data versus on beyond-K-5 data makes no difference; GRPO cannot transport material pretraining never supplied.
A tokenizer ablation quantifies the arithmetic edge: with single-digit number tokenization at 2B scale, beyond-K-5 MathCAMPS pass@1 moves from 1.1% to 1.8% for LittleLearner and 4.6% to 10.3% for Unfiltered. The gap survives.
"The pretraining distribution sets the capability ceiling; post-training is an amplifier, not an engine" has mostly rested on indirect evaluation-side evidence. It now has a reusable controlled testbed. For anyone doing RL or post-training, this is a useful bucket of cold water: when GRPO stops working, the bottleneck is often missing pretraining material, not insufficient training intensity. With corpus and model released, calibration, knowledge-injection, and educational-data studies get a sandbox with a known boundary.
The authors state that phenomena like in-context learning may be less pronounced at 5B than at frontier scale, that capabilities do not follow the human curriculum order (multi-digit division scores above single-digit division), and that the developmental framing is a tool for controlling exposure, not a claim of human-like development.
The reading raises more. MathCAMPS was pruned before use: standards with fewer than 30 gold answers dropped, questions whose gold answer appears verbatim in the prompt dropped (those score up to 23x higher), and the entire 6.EE.B.7 standard excluded. The evaluation set is a trimmed version friendly to the regex grader. Beyond-K-5 absolute scores sit at the floor, on the order of 1%, so part of "cannot be moved" reflects task difficulty, though the Jeopardy cliff and the BPB curves support a real boundary. And with 65% of true K-5 text discarded by the filters, in-scope density is thin enough that K-5 performance may understate what this scale of model could do.