Antirez argues LLM hidden states aren't encoders — they only carry what the next token needs
zafstojano · x · 2026-11-04
In a JEPA-vs-LLM debate, antirez argues an LLM's hidden state doesn't hold the model's full representation — only what's needed for the next token, so it's not an encoder. When @zafstojano objected that backprop from token k+100 flows through token k's embeddings, antirez clarified he means the last-position state: past hidden states are causal and never trained for similarity, so they underperform a proper encoder for pair similarity.
Related event: antirez Clarifies LLM Hidden States Are Not Full Internal Representations(3 posts)→