Supervised Fine-Tuning vs. In-Context Learning: An Equilibrium Analysis of LLM Personalization under Congestion
Fengzhuo Zhang, Zhuoran Yang, Dirk Bergemann
cs.LG, econ.TH, stat.ML
2026-07-16
A Yale linear model shows SFT beats ICL only past a coverage-SNR threshold; shared GPU congestion can reverse that ranking, while offering both methods never cuts max profit.
Personalizing a shared LLM usually means one of two moves: drop demonstrations into the prompt (in-context learning, ICL) or update weights on private data (supervised fine-tuning, SFT). SFT burns more GPU time; ICL is cheaper and less flexible. When many users personalize on the same cluster, one user's SFT job lengthens everyone else's wait.
Most economics papers on LLMs stop at token prices or scaling laws and do not separate the two algorithms statistically. Fengzhuo Zhang, Zhuoran Yang, and Dirk Bergemann at Yale treat linear regression as a proxy for LLM behavior, then embed it in a continuum congestion game. They ask which method wins without congestion, what equilibrium looks like when compute is shared, and whether a platform should sell both.
Pretraining is a linear regression whose samples can come from different latent tasks, producing a Gaussian prior. Personalization data share a single target task.
ICL is Bayesian updating of that prior. It only revises beliefs in the subspace the prior supports. Directions missed in pretraining are projected away, leaving an irreducible slab of task variance. SFT is a regularized estimator centered at the pretrained weights; λ keeps the update from wandering too far. λ=0 collapses to least squares on the personalization data; large λ pins the solution at the pretrained point. Unlike ICL, SFT can move the whole parameter space, including unseen directions.
User cost is prediction error plus (resource per sample) times (unit price p plus a wait that grows with aggregate demand R). SFT uses more resource per sample than ICL. The population is a continuum: each user picks an algorithm and a sample size, and R must equal the resources they actually request. The platform is a Stackelberg leader. It sets p first, then earns (p−1)R, where 1 is the normalized cost of one compute unit.
To compare the two methods they impose spectral alignment and isotropy, so error splits into d independent one-dimensional problems. Coverage is r/(d−r), with r the effective pretrained rank and d the task dimension. Signal-to-noise in the uncovered subspace is κ = sτ/(2σ̃²).
Proposition 3.6 is a threshold. Optimally regularized SFT has error no worse than ICL if and only if coverage is at least a critical value Rcrit that depends on κ. When κ≥1, SFT strictly wins for any positive coverage. When 0<κ<1, SFT wins only if coverage is large enough. ICL can win because it refuses to learn in uncovered directions and pays a fixed task variance τ; SFT tries to learn there and pays 2σ̃²/s, which is a bad deal when the data are noisy and scarce. Corollary 3.4: in non-degenerate cases ICL weakly beats using the pretrained model as-is.
Congestion can reverse that ranking. An equilibrium exists, and the congestion level R is unique (Theorem 4.1), even if the strategy profile is not. Comparative statics are often non-monotone.
| Variable | Effect on equilibrium congestion R |
| Prior precision π | Always falls: better pretraining substitutes for online personalization |
| Coverage dimension r | Rise-fall-rise when compute is cheap; monotone increasing when it is expensive |
| Task noise σ̃ | Rises with noise while personalization still pays, then falls once noise deters it |
| Costlier SFT | Rise then fall if ICL is cheap enough; otherwise monotone decline |
| Unit resource price p | Monotone decline (Theorem 5.2) |
The profit-maximizing price is finite under regularity conditions (Proposition 5.3). Theorem 5.5: adding SFT to the menu cannot lower equilibrium congestion or maximal profit. SFT is the heavy option, so anyone who switches into it raises aggregate demand.
They train a 22M GPT-2 (embedding 256, 12 layers, 8 heads) from scratch on a rank-deficient design with coverage r=15 and ambient dimension d in {15, 17, 19, 21}, on noisy linear regression (noise variance 0.5). ICL error falls then plateaus; the plateau scales roughly linearly with uncovered dimension d−r, with R²=0.96. SFT beats ICL at large sample sizes (e.g. 500) and loses near 1 sample; more SFT data can drive error toward 0. That checks the statistical predictions, not the congestion game.
A documentation review of 21 firms that released major foundation models in the AI Index 2024/2025 reports the share offering both SFT and ICL: 9.5% in 2021, 14.3% in 2022, 42.9% in 2023, 66.7% in 2024, 71.4% in 2025. "Offering SFT" means a fine-tuning API; releasing open weights does not count. Apple, DeepSeek, Meta, and Midjourney are listed as not offering one.
The operational rule is simple enough to repeat. If the task already sits in the pretrained prior and demonstrations are few or noisy, ICL is the safer bet. If the data are informative and the task needs knowledge the prior never saw, SFT is the one that can actually learn it. On a shared cluster, wait times rewrite that rule. Broader coverage, harder tasks, and a higher SFT unit cost can each raise total load over some range instead of cutting it.
For a platform the claim is conservative: putting SFT on the menu does not cut max profit, but it can raise load, so price has to pull demand back. That is a mechanism story for why fine-tuning APIs spread from 2021 to 2025, not a causal estimate.
The GPT-2 linear-regression experiment shows that "ICL cannot touch uncovered directions" is not only an algebraic artifact. It shows up on the in-context linear testbed used by Garg et al.
The authors flag two extensions themselves. Real markets have competing platforms; the model has one. Reasoning models trade extra latency for quality, so the algorithm set needs another option.
The bigger gap is evidence. Spectral alignment, isotropy, and a quadratic congestion function are there for closed forms. The appendix checks exponential and hinge waits numerically, with no production traces. The only neural net is a 22M GPT-2 on linear regression: no language tasks, and no closed loop from user choice to queue. The platform table is a document review; excluding open weights marks Meta and DeepSeek as non-providers and only lines up in direction with "adding the menu does not hurt profit." LoRA and adapters never enter the menu. User types are statistical primitives (d, r, σ̃, π, τ), not billing logs.