FactorEngram paper: factorized n-gram memory with basis-level gating beats Engram on 340M/1B models
pmttyji · reddit · 2026-10-06
A new arXiv paper, FactorEngram, improves lookup-based memory scaling for LLMs. Unlike Engram, which stores each retrieved n-gram embedding in its own hashed slot with a scalar gate, FactorEngram retrieves sparsity-regularized coefficients over a shared dictionary of basis vectors, letting related patterns reuse components and letting context gate each memory component individually. It covers both single tokens and multi-token n-grams and systematically studies branch insertion points. On 340M- and 1B-parameter Transformer backbones it improves language modeling and downstream tasks; ablations show inserting the memory branch before the attention sublayer in middle layers works best.
More from Research
- Dessa's dialog-only voice training isn't new: dGSLM did it in 2022 with just 2k hours — rdesh26 · 2026-10-24
- NYU talk argues 30+ years of consciousness research may be partly wrong — birchlse · 2026-10-24
- Psycholinguistic analysis finds Claude Opus 5 the most out-of-distribution model tested — gleech · 2026-10-24
- Google Research tests 180 agent setups: multi-agent hurts sequential tasks by 39-70% but boosts parallel ones 80.9% — ombre-ai · 2026-10-24
- Loopholing: pushing self-conditioning all the way in discrete diffusion language models — NandoDF · 2026-10-24
- Review-it tool suggests 10 best-fit journals and conferences for your paper — Faheem_uh · 2026-10-24