Bayesian Model Selection Explains Why Proxy-Scale Winners Fail at Target Scale

On September 9, Andreas Kirsch (@BlackHC) published a long explainer thread with a full paper, "Bayesian Model Selection and Scaling Laws," addressing an overlooked issue in model scaling: the "best recipe" chosen on a proxy scale may lose on the target scale, because loss curves can cross as scale grows, causing rank reversals—and the three "best" criteria answer different questions.

Confirmed

• Core theoretical framework: for an ideal Bayesian learner, by the chain rule of probability, the negative log evidence equals the area under its loss curve. The three criteria thus each capture something different: validation loss estimates the endpoint of the curve, CLML (the difference in log marginal likelihood) is a tail area, and Bayesian evidence evaluates the entire trajectory.
• Quantitative results: with equal loss floors, the three criteria separate models at different rates (with Δλ the difference in model complexity). The endpoint gap is about Δλ/N and vanishes asymptotically—corresponding to the inconsistency of LOO-CV (citing Shao 1993); the tail area and total area have distinct separation properties, with the tail area retaining growth signals.
• Bayesian evidence can recover dimensional information that held-out loss misses (e.g., in the Bayesian PCA setting, citing Lotfi et al. 2022), but only under equal-data comparisons.
• Loss-curve crossings are shaped by three mechanisms: model misspecification sets the loss floor; model complexity sets the late-stage 1/n excess error; prior fit and strength set early behavior.
• Scholarly lineage: tail-area criteria date back to research in the 1990s; the CLML difference between two models is a log partial Bayes factor, and the trade-offs identified back then still hold today.
• Practical caveat: SGD does not inherit the Bayesian evidence identity; the cumulative sum of per-epoch losses depends on the specific pipeline and data order, and is in essence a prequential score rather than Bayesian evidence—but the geometric structure of the curves and the analytical approach can be transferred from the Bayesian case to ordinary training.

Why it matters

Scaling-law experiments typically pick recipes on small proxy runs and scale up; this work systematically explains, from a Bayesian model-comparison perspective, when and why such extrapolation fails, providing theoretically grounded criteria for recipe selection in scaling-law research and helping avoid the wasted resources of "proxy-scale winners flopping at target scale."

2026-09-09 ~ 2026-09-09 · 10 related posts

• Bayesian model selection explains why proxy-scale winners can lose at target scale — BlackHC · 2026-09-09
• [source] Loss curves cross as you scale: Bayesian model selection explainer shows three 'best' criteria pick different winners — BlackHC · 2026-09-09
• [source] Why these three criteria: negative log evidence equals the area under the loss curve — BlackHC · 2026-09-09
• Why loss curves cross: misspecification, complexity, and prior fit — BlackHC · 2026-09-09
• SGD doesn't inherit the Bayesian identity — summed losses are a prequential score — BlackHC · 2026-09-09
• Model complexity gap drives three separation rates; LOO-CV is inconsistent — BlackHC · 2026-09-09
• Three selection criteria separate at different rates — LOO-CV is inconsistent — BlackHC · 2026-09-09
• Bayesian evidence recovers dimensionality where held-out loss fails — at equal n — BlackHC · 2026-09-09
• Thread follow-up: the tail-area criterion traces back to 1990s Bayesian literature — BlackHC · 2026-09-09

1 near-duplicate retellings: BlackHC