antirez: LLM last-position hidden states have inherent limits for similarity
antirez · x · 2026-11-04
antirez clarified he was referring to the hidden state at the last position — the one producing the next token. He noted that while people pool hidden states from past positions for more context, those states are causal and their representation shape was never trained for similarity, making them inherently limited compared to a proper encoder for pairwise similarity tasks.
Related event: antirez Clarifies LLM Hidden States Are Not Full Internal Representations(3 posts)→
More from Research
- Axiom ships 16M-line AI formalization of the Stacks Project, countering 'AI code is slop' — Scobleizer · 2026-11-05
- GRETA platform screens 1,000 FDA drugs, finds statins may reverse IPF signature — danielmckinn0n · 2026-11-05
- TLAPS-Bench author clarifies: passing formal verification doesn't mean bug-free implementations — moarbugs · 2026-11-05
- Swyx digs into MIT CSAIL's viral 'Weight-Space Programming': cool, reproducible, but temper the hype — rschu · 2026-11-05
- Visual Token Routing cuts VLM agent visual tokens up to 5.8x at same accuracy — wellecks · 2026-11-05
- Matthew Berman on weight-space programming: a step toward recursive self-improvement — Matthew Berman · 2026-11-05