Loss curves cross as you scale: Bayesian model selection explainer shows three 'best' criteria pick different winners
BlackHC · x · 2026-09-09
In an educational thread, BlackHC explains why recipes chosen as winners at a proxy scale can lose at the target scale, via Bayesian model selection and scaling laws.
Key points:
- Loss curves can cross as scale grows, reversing rankings
- Three 'best' criteria answer different questions: best frozen model at a budget → final height; best hypothesis for the whole stream → total area; best learner after a warm start → tail area (CLML)
- For an ideal Bayesian learner, negative log evidence equals the area under the loss curve; validation loss estimates the endpoint, CLML is a tail area, and evidence scores the whole trajectory
- The criteria disagree only when curves cross, flipping in a fixed order: the endpoint flips at the crossing, while tail and total areas flip later once later gains repay the earlier deficit
- Practical advice: fit across budgets and seeds, and report uncertainty on the crossing
An exact Bayesian regression toy setting demonstrates three different winners under the three criteria at different scales.
More from Research
- Open-source decision models sold their own base in Red Alert 2 — then learned until no human could beat them — heathercmiller · 2026-10-27
- DeepMind paper: LLMs can't sample randomly on their own — pass random numbers from code instead — rohanpaul_ai · 2026-10-27
- World Model Assigns Weight and Friction to Objects, Then Simulates with a Physics Engine — jonstephens85 · 2026-10-27
- Token Collapse: Study Finds Graph Soft Prompting Tokens in LLMs Fail to Encode Structure — Sapienza · 2026-10-27
- Mathathon 2.0 launches with Terence Tao's SAIR, $1.5M in compute for teams — _sathvikr · 2026-10-27
- The 'sweet lesson' is real: two kinds of ML problems need opposite strategies — mike64_t · 2026-10-27