79K-param RLT test: learns state tracking but fails on longer sequences while GRU holds
mike64_t · x · 2026-09-14
A small-scale reproduction of RLT (79K params, 3 seeds, trained at 32 steps) on two synthetic state-tracking tasks shows final-state accuracy of 100% at length 32, but dropping to 60.8%/20.7% at length 128, while a GRU holds 100% at both lengths.
A respondent offers deeper analysis: traditional recurrent cells have an unreasonably good inductive bias for synthetic state tracking—they can literally express the required computation and learn it exactly under teacher forcing, whereas transformers must construct approximations. He argues comparing this to reasoning-like computation is wrong, that pre-training loss gains are the wrong benchmark angle for recurrent models, and that reasoning is recall-heavy—tied depth is simply a very different regime from distinct-parameter depth.
More from Models
- Dev complains Claude feels weird despite great evals, blames post-training quirks — MaziyarPanahi · 2026-09-14
- Developer complains Claude Opus 5 'got dumber' this week — draginol · 2026-09-14
- Zhipu raises $5B, with 60% earmarked for next-gen GLM models and a self-training RSI loop — teortaxesTex · 2026-09-14
- Writer gives up on Opus: made-up jargon feels like 'mild gaslighting' — julianharris · 2026-09-14
- Ollama's jmorgan: small models now handle most conversational and reasoning use cases — ollama · 2026-09-14
- GLM Flash 5.3 keeps slipping into Chinese mid-conversation, user reports — DevDminGod · 2026-09-14