ML engineer corrects antirez: last-token residual state isn't optimized only for next-token prediction
mlpowered · x · 2026-11-06
Responding to antirez's claim that the last token's residual state is optimized solely for next-token prediction, mlpowered explains that during training, next-token prediction gradients at every position flow back through attention to all prior residual states — meaning these states can encode information useful many tokens ahead, as supported by cited papers. antirez had argued the last-token state lacks embedding-style information without pooling; this exchange is a notable technical debate on whether LLM hidden states can serve as embeddings directly.
Related event: antirez debates ML engineers on whether LLM hidden states work as encoders(9 posts)→
More from Models
- Claude Models Send Your Email to Websites—mitsuhiko Traces It to Cargo.toml — mitsuhiko · 2026-11-06
- Not Claude Code: the Claude model itself leaks your email from Cargo.toml, finds mitsuhiko — mitsuhiko · 2026-11-06
- Perplexity cuts Agent API prices across effort levels, adds new ultra tier for long-running tasks — denisyarats · 2026-11-06
- GPT Image 2.6 tops Image Arena, up to 2.6x faster, redefines Pareto frontier — cantrell · 2026-11-06
- FLUX 3 models beat FLUX.2 in all 19 Text-to-Image categories; Klein tops $9/1k price frontier — ArtificialAnlys · 2026-11-06
- Budget-bound dev weighs Kimi subscription against a second Claude account — mohitaksh · 2026-11-06