What Has a Foundation Model Found? Using Inductive Bias to Probe for World Models
Keyon Vafa, Peter G. Chang, Ashesh Rambachan, Sendhil Mullainathan
ICML 2025
cs.LG, cs.AI
2025-07-09
A 109M transformer predicts orbits at R²>0.9999 but recovers five fake gravity laws; Othello models play legal moves ~100% of the time with weak board-state inductive bias.
The sales pitch for foundation models is old in spirit: get sequence prediction right, and deeper structure should fall out. Kepler could place planets in the night sky with striking precision. Newton later extracted a force law that also explained tides and cannonballs. If a network predicts the next planetary position almost perfectly, is it still a Kepler machine, or has it crossed over to Newton?
Existing tests do not match how foundation models are used. Linear probes and SAEs ask whether a state can be read from activations; readability does not mean the model uses that state on a new task. Legality of the next move, or R² on the training objective, only scores the original job. In practice the model is given a small new dataset and asked to adapt. The authors want to know whether that adaptation's inductive bias lines up with a postulated world model.
They call the procedure an inductive bias probe. Fix a candidate world model, a map from inputs to states. Repeatedly fit the foundation model on tiny synthetic datasets whose labels are functions of that state, then inspect how it extrapolates to held-out inputs.
For finite states and binary labels this splits into two scores, 1 perfect and 0 no better than noise:
Either score alone can be gamed. A constant predictor aces R-IB and zeros D-IB. For continuous states they plot a calibration curve. An oracle fits the allowed function class on the true state; the model's extrapolative predictability should track the oracle's as a function of state similarity. Sitting on the 45-degree line means an inductive bias toward that world model.
Three domains. Orbital mechanics: Newtonian two-body trajectories around a sun, planet-planet forces dropped, 10M sequences (about 20B tokens), a 109M transformer trained 25 epochs to predict the next position. Coordinates are quantized into 7K bins per axis over [-50, 50] AU. The postulated state is masses, relative positions, and relative velocities. They then fine-tune the same model to predict force vectors and run PySR symbolic regression on the predictions to recover the implied gravity law. Lattice: an agent on a 2-to-5-site line with boundary-illegal moves. Othello: 20M games, true state the 8x8 board. Architectures: RNN, LSTM, Transformer, Mamba, Mamba-2, each compared with an untrained run that starts from random init on the synthetic tasks. An appendix also queries o3, Claude Sonnet 4, and Gemini 2.5 Pro in context: 5 solar systems, 450 steps each, 10 labeled forces (about 2%), with no hint that the labels are forces.
The orbital model is a superb Kepler engine. Held-out R² exceeds 0.9999. Rolled out from step 50, 1-step MSE is (1.90±0.45)×10⁻⁸ against a previous-position baseline of (1.16±0.21)×10⁻⁴; at 100 steps it is still (3.74±3.37)×10⁻⁵ versus (4.04±0.47)×10⁻².
It is not a Newton engine. Fine-tuned force arrows miss the true field. Symbolic regression on five galaxy slices of predicted force magnitudes returns five different formulas, none of them F ∝ m1 m2 / r². Galaxy 2 yields F ∝ cos(cos(2.19 m1)); Galaxy 4 yields F ∝ sin(r + 8569.2 + 1/m1). The same regression on an oracle that sees true state recovers the real law in every slice.
Lattice and Othello show the same crack. Next-token legality is 1.00 for every architecture on the 5-state lattice, and 0.992 (RNN) to 0.999 (Transformer, Mamba) on Othello. Inductive bias is another story. After next-token pretraining on the 5-state lattice, the transformer scores R-IB 0.483 and D-IB 0.677; LSTM scores 0.782 and 0.921. As the number of states grows, IB falls for every model and the transformer falls fastest. On Othello, NTP models sit around R-IB 0.56–0.70 and D-IB 0.61–0.80.
Models with stronger IB also transfer better to deterministic functions of the board. Majority Tiles accuracy: 0.754 from a randomly initialized transformer, 0.956 after NTP, 0.970 for NTP Mamba-2. Unsigned correlation between transfer metrics and the summary ratio R-IB/(1−D-IB) reaches 0.960 on Edge Balance accuracy.
What the models extrapolate from is not the full board. It is the set of legal next moves. Splitting D-IB by whether two distinct states share that set, D-IBq= is lower than D-IBq≠ for every architecture; on Othello the transformer is 0.458 versus 0.625. Fine-tuning to reconstruct the board often gets the stones wrong while matching the legal-move set.
Frontier LLMs do not recover Newton either. Symbolic regression on o3 and Gemini 2.5 Pro both collapse to F ∝ m1; Claude Sonnet 4 yields F ∝ 1/(m2−0.50).
High pretraining scores are not a world model, at least on these domains where ground-truth state is known. Legal next moves, trajectory R², and linearly readable state do not guarantee that adaptation will respect that state. Anyone hoping an orbital foundation model will drop onto forces, energies, or new boundary conditions should run this probe first.
On Othello the IB numbers move with transfer, so the probe is at least a useful checkpoint ranking, more aligned with "does it use the board" than another legality leaderboard.
Do not inflate this into a verdict on ChatGPT. It is a diagnosis on controlled sequence worlds: next-token pretraining here behaves like a bag of task heuristics, especially the coarsening that groups states with the same legal next tokens, not a compact reusable world model.
The largest caveat is stated by the authors. The probe needs a world model specified in advance. It can reject Newton and reject the board; it cannot name the representation the network actually uses. They test one alternative, the next-token partition, and leave automatic discovery of the implicit world model as future work.
The data are also unusually clean. Orbits are generated from Newton with inter-planet forces omitted; 7K-bin coordinates may themselves invite geometric shortcuts. The LLM study uses five solar systems and ten in-context labels, and the authors note that the recovered expressions being simpler may just reflect the smaller setup. Foundation model here mostly means a pretrained-then-finetuned learner, not an open-weight general chatbot.
R-IB and D-IB also move with the sampling distribution and the fine-tuning protocol. In the appendix, raising Othello fine-tune examples from 10 to 500 drops R-IB and lifts D-IB, so the absolute numbers are protocol-sensitive. There is no head-to-head table of this probe versus a standard linear probe on the same models and states, only a conceptual contrast.