Fractale-350M: Open-Sourcing Memory as a Training Behavior
KKuettes · reddit · 2026-07-20
Independent researchers have released the Fractale-350M-base model, exploring a novel long-term memory mechanism by treating memory as a training behavior instead of relying on long contexts.
Core Design
- Memory Bank Mechanism: The model's only long-term memory is a bank of 8 vectors. It writes a summary vector every 512 tokens and uses FIFO (First-In-First-Out) to discard the oldest memory.
- Hypernetwork Reading: Vectors in the memory bank are expanded into low-rank MLPs via a hypernetwork, participating directly in the forward pass, bypassing traditional text retrieval or attention mechanisms.
- Pre-training Objective: Forces the model to rely solely on these 8 memory vectors to predict the beginning of the next unseen text chunk.
Experimental Data
- Memory Dependence (GAP): On code datasets, the cross-entropy difference between retaining and resetting memory reaches up to 9.4 nats (CE drops from 12.9 to 3.45); for web data, it is 7.3 nats. This indicates the model's deepening reliance on its own memory during training.
- Mechanism Validation: Smaller-scale controlled experiments confirmed that memory content can be addressed via cues, survives after 2000+ steps, and transmits information bidirectionally (e.g., between docstrings and code).
Open Source & Limitations
- This is a base model; it is not instruction-tuned or aligned, and its raw fluency aligns with expectations for 386M parameters (trained on 10 billion tokens).
- The model only memorizes "key points" (like domains, structures, facts) and cannot recite source text verbatim.
- Fully Open-Sourced: Includes model, code, complete research logs, and paper. Total pre-training cost was roughly $320 (8xA100 rental).
The author plans to use instruction tuning and reinforcement learning in the second phase to teach the model to use these memories actively and deliberately.
More from Models
- GPT-5.6 writes well but is instantly forgettable, user complains — BasedRaddka · 2026-09-11
- GPT-6 Astra beats Factorio with enemies in 44 in-game hours at ~$4,500 API cost — liminal_bardo · 2026-09-11
- awesome-llm-leaderboards: an open-source directory of LLM leaderboards, pricing tables, comparison tools — Last_Establishment_1 · 2026-09-11
- Anthropic claims it works to keep eval environments unidentifiable to models — MaxKannen · 2026-09-11
- Nex N2.5 Pro released on Hugging Face with 407GB of weights — jinnyjuice · 2026-09-11
- RoMa v2 image matching model unveiled in the usual black poster — ducha_aiki · 2026-09-11