Follow-up: gradient flow lets residual states encode information for far-future predictions
mlpowered · x · 2026-11-06
The follow-up half of mlpowered's exchange with antirez: because next-token prediction gradients flow back through attention to all prior residual states during training, these states can be updated to aid predictions many tokens ahead, supported by the papers he linked — further rebutting the claim that the last-token residual state is optimized only for the next token.
Related event: antirez debates ML engineers on whether LLM hidden states work as encoders(9 posts)→
More from Models
- Claude Models Send Your Email to Websites—mitsuhiko Traces It to Cargo.toml — mitsuhiko · 2026-11-06
- Not Claude Code: the Claude model itself leaks your email from Cargo.toml, finds mitsuhiko — mitsuhiko · 2026-11-06
- Perplexity cuts Agent API prices across effort levels, adds new ultra tier for long-running tasks — denisyarats · 2026-11-06
- GPT Image 2.6 tops Image Arena, up to 2.6x faster, redefines Pareto frontier — cantrell · 2026-11-06
- FLUX 3 models beat FLUX.2 in all 19 Text-to-Image categories; Klein tops $9/1k price frontier — ArtificialAnlys · 2026-11-06
- Budget-bound dev weighs Kimi subscription against a second Claude account — mohitaksh · 2026-11-06