ML Professor Fleuret: Transformers Minimize Cross-Entropy Without Understanding a Thing
francoisfleuret · x · 2026-11-08
François Fleuret, ML professor at University of Geneva, sarcastically describes how Transformers work: "I have no idea what this is about, but I minimize cross-entropy better because I leverage every single statistical cue in my flawless recall of the conversation."
His underlying point: we need a model that learns in spite of its inductive bias, not thanks to it — and we should "stop being cross-entropy fetishists." A sharp critique of the limits of the next-token prediction paradigm.
Related event: Geneva Professor Mocks Transformers as Statistical Pattern Matchers(2 posts)→
More from Research
- LiveFoley generates real-time video-to-audio with 156ms latency on one GPU — victor_explore · 2026-11-08
- Diffusion Models That Generate Neural Network Weights: Community POC 'nngen' Released — Specific-Tax-6700 · 2026-11-08
- Why go to conferences when AI reads and writes the papers? A UIST researcher's reflection — plopesresearch · 2026-11-08
- Early verdict: AI-oriented APIs aren't worth it — training-distribution similarity matters more — teortaxesTex · 2026-11-08
- OpenAI AI-generated manuscript claims proof of local smoothing conjecture in all dimensions — jiqizhixin · 2026-11-08
- Library of Layers: Transformers That Reorder Their Own Layers at Runtime Match Baselines — LotfiSanae · 2026-11-08