Memory-native MoE MemT 2.0 (293B) keeps memory inside the forward pass, no retraining needed

MaterialButterfly135 · reddit · 2026-10-19

• Approach: MemT 2.0 is a 293B memory-native MoE model that compresses history into a persistent internal memory state, read via memory attention and updated within a normal forward pass — no gradients, no retraining. Memory write/retain/update/read happen inside the computation, not in an external framework.
• Motivation: on long coding tasks, project constraints, prior decisions and corrections vanish when sessions end; long context and RAG re-supply information rather than actually remembering it.
• Evaluation: internal Coding and Working test suites, budget- and environment-matched against a baseline. The authors stress it's one internal comparison, not a promise; no public leaderboard numbers yet, raw traces to be published.
• Availability: 36B and 293B initial training finished on Hugging Face; they ask whether a trusted public benchmark, a self-hostable size, or a reproducible ablation would make people try it.

Original post →

More from Models

• Together AI teases mysterious 'Space Bunny' launch — togethercompute · 2026-10-19
• Pangram AI detector flags The Atlantic's human-written cover story as 100% AI — nitarshan · 2026-10-19
• Can a world model run in real time on just 12GB of VRAM? — yehiaserag · 2026-10-19
• MIST Framework Separates Execution from Memory to Diagnose Video World Model Failures — hkuhk · 2026-10-19
• Reddit test finds Nemotron 3.5 Super VL best at obscure knowledge, beating GPT-OSS and Qwen — Kahvana · 2026-10-19
• Thought experiment: if billion-dollar companies stopped open-weight releases for a year — ForsookComparison · 2026-10-19

Models channel →