79K-param RLT test: learns state tracking but fails on longer sequences while GRU holds
mike64_t · x · 2026-09-14
A small-scale reproduction of RLT (79K params, 3 seeds, trained at 32 steps) on two synthetic state-tracking tasks shows final-state accuracy of 100% at length 32, but dropping to 60.8%/20.7% at length 128, while a GRU holds 100% at both lengths.
A respondent offers deeper analysis: traditional recurrent cells have an unreasonably good inductive bias for synthetic state tracking—they can literally express the required computation and learn it exactly under teacher forcing, whereas transformers must construct approximations. He argues comparing this to reasoning-like computation is wrong, that pre-training loss gains are the wrong benchmark angle for recurrent models, and that reasoning is recall-heavy—tied depth is simply a very different regime from distinct-parameter depth.
More from Models
- Leak: Google Reportedly Has Another 'Hella Chonky' Model After Carbon — iruletheworldmo · 2026-10-30
- Users report across-the-board GPT quality drop: weaker writing and broken instruction following — Kyrannio · 2026-10-30
- Leak claims Gemini Argon hits Pro trial from November 2nd (unverified) — Emotional-Company847 · 2026-10-30
- OpenAI retiring waves of legacy models: GPT-5 family dies Dec 11, GPT-4.1 by Jan 2027 — ThePeterMick · 2026-10-30
- DeepMind CEO Demis Hassabis tells Musk: testing with trusted partners, coming soon — demishassabis · 2026-10-30
- Adobe stands firm on not training on customer data, using licensed stock instead — omooretweets · 2026-10-30