Bayesian Model Selection Explains Why Proxy-Scale Winners Fail at Target Scale

On September 9, Andreas Kirsch (@BlackHC) published a long thread and full article "Bayesian Model Selection and Scaling Laws," explaining that recipes chosen as "best" at proxy scale can lose at target scale because loss curves cross as scale grows, and three notions of "best" answer different questions. Gerard Sans amplified it the next day.

Confirmed

• Core framework: for an ideal Bayesian learner, negative log evidence equals the area under the loss curve; validation loss estimates the final height, CLML is a tail area, and Bayesian evidence evaluates the whole trajectory.
• Quantitatively, with equal loss floors the three criteria separate models at different rates (Δλ = complexity gap): final-height gaps Δλ/N vanish, matching LOO-CV inconsistency (Shao 1993); tail areas retain growth signals while total area can grow unboundedly.
• Bayesian evidence recovers dimensional information where held-out loss fails (Bayesian PCA, Lotfi et al. 2022), but only under equal-data comparisons.
• Curve crossings are shaped by misspecification (loss floor), complexity (late 1/n excess error), and prior fit/strength (early behavior).
• SGD does not inherit the Bayesian evidence identity; cumulative first-epoch losses are prequential scores, though the geometric analysis transfers.
• The author also criticized current data curation as built for older neural networks, ignoring token-isomorphic structure and the co-occurrence geometry Transformers learn.

Why it matters

The work systematically explains when and why proxy-scale extrapolation in scaling-law experiments fails, providing principled criteria distinctions that can prevent wasted compute on recipes that flip rankings at target scale; it also signals that data curation needs redesign for Transformers.

2026-09-09 ~ 2026-09-10 · 12 related posts

• Bayesian model selection explains why proxy-scale winners can lose at target scale — BlackHC · 2026-09-09
• Loss curves cross as you scale: Bayesian model selection explainer shows three 'best' criteria pick different winners — BlackHC · 2026-09-09
• [source] Why these three criteria: negative log evidence equals the area under the loss curve — BlackHC · 2026-09-09
• Why loss curves cross: misspecification, complexity, and prior fit — BlackHC · 2026-09-09
• SGD doesn't inherit the Bayesian identity — summed losses are a prequential score — BlackHC · 2026-09-09
• Model complexity gap drives three separation rates; LOO-CV is inconsistent — BlackHC · 2026-09-09
• Three selection criteria separate at different rates — LOO-CV is inconsistent — BlackHC · 2026-09-09
• [source] Bayesian evidence recovers dimensionality where held-out loss fails — at equal n — BlackHC · 2026-09-09
• Thread follow-up: the tail-area criterion traces back to 1990s Bayesian literature — BlackHC · 2026-09-09
• Why winning training recipes at proxy scale can lose at target scale — gerardsans · 2026-09-10
• Data curation has massive blindspots: Transformers learn co-occurrence geometry, not features — BlackHC · 2026-09-10

1 near-duplicate retellings: BlackHC