15M-param face-off: linear RNN ties Transformer on accuracy, trains ~6-7x faster
AlarmingTrouble5261 · reddit · 2026-10-15
Inspired by TypeSafe's Jev (a non-generative model outputting only bounded decisions), the author ran a budget-matched (±0.02%, 15.09M params) head-to-head between a linear recurrent backbone (their Gated Segmented State Space) and a standard attention backbone with an identical decision head, on a long-context retrieval-decision task (choose which of 4 sentences appears in a 2k–8k char passage). Everything trained on a Colab T4, fully reproducible, repo open-sourced.
Results:
- Accuracy: noise — 26.8% vs 28.2% on test (chance 25%). Neither learns much; the negative result is the point: small decision models are accuracy-starved regardless of substrate.
- Latency: a real crossover — Transformer is faster below 1550 tokens; at T=8192 the linear model is 2.37x faster (108.6 vs 257.7 ms), O(T) vs O(T²) showing up where theory says it should.
- Training cost: the big gap — 6 finetune epochs took 30 min for the linear model vs 40–45 min per epoch for the Transformer, a 6–7x gap that grows with scale.
- Earlier rounds (V2/V3, shorter decision tasks) had the linear model winning outright on accuracy (50.4 vs 46.2, 66.2 vs 61.0).
Takeaway: linear recurrence wins or ties short-context decisions and dominates long-context cost; it just doesn't beat attention on the hard retrieval task at 15M. A bidirectional V5 is currently running.
More from Research
- RFD4-Proteina's 'underrated' docs detail partial diffusion for protein redesign — AllThingsApx · 2026-10-23
- MPI-Sec PhD Thesis on Vulnerability Detection Benchmarks Includes Pre-LLM Classic "Top Score on the Wrong Exam" — mboehme_ · 2026-10-23
- LiveNerf's 29-day data finds no nerfing of Claude Opus 5.5 — TheOnlyVibemaster · 2026-10-23
- New paper makes 9 falsifiable predictions toward a major AI-enabled terror attack by 2027-2028 — StephenLCasper · 2026-10-23
- HumanoidPose recovers humanoid robots' full-body joint states from RGB video alone — cholz · 2026-10-23
- Sophont's Holon-S1 predicts genes and biomarkers from routine pathology slides — iScienceLuvr · 2026-10-23