Fleuret: Teacher Forcing Trades Temporal Iteration for Depth in Transformers
François Fleuret explains that teacher forcing lets transformers convert temporal iteration over a medium-sized state into depth-wise iteration over a huge state, pleasing GPUs, but at the cost of learning internal state dynamics, leaving only token-by-token reasoning.
2026-10-24 ~ 2026-10-24 · 2 related posts
- Fleuret: teacher forcing turns time iteration into depth iteration, and GPUs love it — francoisfleuret · 2026-10-24
- Fleuret: teacher forcing leaves no room for internal state dynamics, hence clunky token reasoning — francoisfleuret · 2026-10-24