Training, learning and inference: unified dynamics of neural systems
Mian Wang
cs.LG
2026-08-21
A GFG study on nanoGPT predicts capability flips at 91.43% with a second-order margin model. Concentrated RL feedback cuts unreinforced-skill accuracy by 39.06 points.
Training, learning, and inference are three jobs in a stack: backprop writes parameters, capability shows up on a eval split, inference reads a frozen net. Mechanistically they are often told as separate stories. Lineage, OpenTelemetry, and Autograd each keep one projection of how a result was formed. This paper wants a fuller record of one concrete generation, then uses that record to ask how a real parameter update changes capability and how a frozen network later spends that change.
It is not a new optimizer or architecture. It is a causal-intervention account of training dynamics, run mainly on nanoGPT.
An atomic generation fact is a five-tuple: the sources that actually took part, the transformation that ran, the concrete occurrence, the outcome or an explicit discard, and the relation role. Those facts compile into a Generation-Fact Graph (GFG). Later analysis, intervention, replay, and validation write new facts onto the same history.
On nanoGPT, training is the evolution of a parameter-optimizer system with state and memory. A realized update enters the current receiving state (parameters plus Adam), and the functional response is jointly set by that state and by target-specific update geometry. The response is finite-amplitude and nonlinear. Learning is the lasting reallocation of distributed functional support caused by those responses. Capability appears only when a target's internal state crosses its readout boundary.
The predictor runs after Δθ exists and before the target forward pass. It combines the current margin with first- and second-order responses to the update, then asks whether the predicted post-update margin stays positive. The four labels are remain-correct, correct-to-incorrect, remain-incorrect, and recover. It sees Δθ. It does not see post-update outputs.
On the inference side, component gating and version rollback swap a trained component for its pre-formation version and check whether logits and accuracy fall, then whether restore brings the exact logits back. A reinforcement campaign varies the concentration of correct positive feedback and watches target support versus other-skill margins.
Eight training-learning experiments form the chain. Receiving-state exchange shows that the same Δθ produces different functional responses on different states. Sweeping a realized update across amplitudes α ∈ {0, 0.125, …, 1} yields saturation, acceleration, turnback, and sign reversal; local first- or second-order fits often miss the endpoint. Across 72 realized-update slices, support-reallocation magnitude correlates 0.881 (Spearman) with update magnitude and 0.765 with absolute capability change. Mean reallocation is 0.2080 when capability changes and 0.0340 when it does not. A single component's update size does not reliably say whether that component later gains or loses responsibility.
On the held-out confirmation split the second-order predictor is right on 4652 of 5088 cases: 91.43% target-boundary accuracy, 92.17% balanced accuracy, 91.49% macro recall. Per-class recall: remain-correct 87.18%, correct→incorrect 98.91%, remain-incorrect 95.90%, recover 83.95%. Across 12 full runs it is 14069 / 15264 (92.17%).
On 13 nanoGPT histories and 52 checkpoints, gating changes the full logits; 23 target groups produce 23 distinct support patterns; all 138 pairwise gatings are non-additive; all 52 rollbacks drop accuracy and restore recovers the original logits. In the RL campaign, 12/12 seeds are dose-ordered: exclusive versus balanced feedback raises target support share by 9.67 points and cuts other-skill accuracy by 39.06 points. Rebalancing returns other-skill accuracy to 99.48% with the target still at 100%. ResNet/CIFAR and diffusion/CIFAR, three seeds each, recover receiving-state-conditioned responses, support reorganization, and frozen projection; the diffusion side adds 504 independently checked response records.
For people who debug training, this splits "the loss went down" into three inspectable pieces: how close a target sits to its boundary, the geometry of this update for that target, and the Adam receiving state. 91% transition accuracy says that, at least in this nanoGPT system, capability flips are not only visible after the fact. The RL numbers are more concrete: concentrated positive feedback pulls support toward the reinforced skill and collapses neighboring margins; spreading the feedback later recovers them. That is a reward-overfit warning measured as support share.
It will not drop into a training script. The predictor fires after Δθ is already computed, so it saves a target forward, not a backward pass. The GFG ontology is heavy, and the main text barely lines up against influence functions or other training-dynamics work.
Single author, nanoGPT-centric. Cross-system checks stop at a ResNet classifier and a CIFAR diffusion model: no large LM, no standard NLP benchmark. The predictor uses the real Δθ plus first- and second-order margin terms, so 91% tests whether those coordinates suffice, not whether the next gradient can be known early. The Attention section reverse-engineers "query-conditioned projection plus joint combination"; it does not ablate new attention variants. The RL study varies feedback concentration, not a full GRPO or PPO loop. Failure cases are thin: where the 8.57% prediction errors sit, and how far a small-model result travels, are mostly unreported.