Do hidden states encode beyond next-token info? A researcher answers antirez
burny_tech · x · 2026-11-06
Responding to antirez, researcher mlpowered addresses two questions: (1) Does a hidden state at token T encode more than what's needed for T+1? Unambiguously yes — the architecture allows it and there are concrete examples. (2) How often does it happen? That's an empirical question, and in practice it occurs frequently, not as an anomaly.
More from Research
- NEJM AI randomized trial: predictive algorithm catches autoimmune diseases years earlier — EricTopol · 2026-11-06
- LessWrong essay urges AI safety researchers to reconcile contradictory evidence instead of firing off experiments — ChenhaoTan · 2026-11-06
- Axiom Math releases Lean formalization of the Weil conjectures (I-IV) — soumitrashukla9 · 2026-11-06
- Measuring factuality in AI search: sentence-level auto-raters and the FACTS benchmark — lilyraynyc · 2026-11-06
- NVIDIA Releases Four Open-Weight Cosmos3-Nano World Models for Driving and Robot Simulation — liu_mingyu · 2026-11-06
- CoRL poster preview uses a cooking analogy for synthetic vs human robot data — jsuarez · 2026-11-06