Entropy Collapse May Not Need SFT Reinsertion; Souping in Pretraining Works Surprisingly Well
ar0cket1 · x · 2026-10-07
@ar0cket1 argues that if entropy collapse itself were fixed (sharing an entropy curve plot), the SFT reinsertion might be unnecessary; he's surprised how well souping in pretraining worked, but thinks the critic seemed unnecessary and probably didn't help.
A companion take on the same training postmortem: diagnose entropy collapse first before adding recipe complexity.
More from Research
- QUEEN paper distills AlphaZero-style chess search into language for LLMs — danqi_chen · 2026-10-07
- COLM paper asks whether VLMs can internalize tool calls in latent space instead of calling them — PMinervini · 2026-10-07
- CLeaR 2027 Opens Its Submission Portal — ArthurGretton · 2026-10-07
- MIT paper: a minimalist agent loop that passes history as code variables beats Letta and ACE at half the cost — rohanpaul_ai · 2026-10-07
- doodlestein releases explainer video alongside open-source companion repo — doodlestein · 2026-10-07
- A 3Blue1Brown-Style Explainer Rendered End-to-End in Rust by franken_manim — doodlestein · 2026-10-07