NextLat: a tiny training-objective tweak has GPT predict its own next hidden state
yacinelearning · x · 2026-10-29
In a chat with MIT's Jayden Teoh, Yacine breaks down NextLat (next-latent prediction), a minimal fix to the training objective the then-undergrad worked on, targeting where next-token transformers still fall short (e.g., path-star graph problems):
- Bolt a small MLP onto a GPT and have it predict the transformer's own next hidden state
- To do this, internal representations must become belief states — compact summaries of everything relevant to the future
- The MLP can be discarded at inference, adding no deployment cost
- Fun contrast: transformers crush problems that look harder while failing on path-star
Livestream Q&A was sponsored by Arcee AI.
More from Research
- Information-theoretic scaling laws predict memorization and FID in diffusion models — LucaAmb · 2026-10-29
- ICLR reviewer hands out scores of 1,1,1,1 and 2 across five papers — chhaviyadav_ · 2026-10-29
- kalomaze calls out hnet baseline: effective depth unmatched, no wall clock reported — kalomaze · 2026-10-29
- AVERT Study Launched: Can Autonomous Vehicles Reduce Trauma? — zakkohane · 2026-10-29
- Bad binder design drove only ~3% of Phase 2/3 failures — the rest elude AlphaFold — clarejtbirch · 2026-10-29
- Demo shows emergent 'in-context tokenization' beating fixed tokenizers — willdepue · 2026-10-29