Fractale-350M: Open-Sourcing Memory as a Training Behavior
KKuettes · reddit · 2026-07-20
Independent researchers have released the **Fractale-350M-base** model, exploring a novel long-term memory mechanism by treating memory as a **training behavior** instead of relying on long contexts. ### Core Design - **Memory Bank Mechanism**: The model's only long-term memory is a bank of 8 vectors. It writes a summary vector every 512 tokens and uses FIFO (First-In-First-Out) to discard the oldest memory. - **Hypernetwork Reading**: Vectors in the memory bank are expanded into low-rank MLPs via a hypernetwork, participating directly in the forward pass, bypassing traditional text retrieval or attention mechanisms. - **Pre-training Objective**: Forces the model to rely solely on these 8 memory vectors to predict the beginning of the next unseen text chunk. ### Experimental Data - **Memory Dependence (GAP)**: On code datasets, the cross-entropy difference between retaining and resetting memory reaches up to **9.4 nats** (CE drops from 12.9 to 3.45); for web data, it is 7.3 nats. This indicates the model's deepening reliance on its own memory during training. - **Mechanism Validation**: Smaller-scale controlled experiments confirmed that memory content can be addressed via cues, survives after 2000+ steps, and transmits information bidirectionally (e.g., between docstrings and code). ### Open Source & Limitations - This is a **base model**; it is not instruction-tuned or aligned, and its raw fluency aligns with expectations for 386M parameters (trained on 10 billion tokens). - The model only memorizes "key points" (like domains, structures, facts) and cannot recite source text verbatim. - **Fully Open-Sourced**: Includes model, code, complete research logs, and paper. Total pre-training cost was roughly $320 (8xA100 rental). The author plans to use instruction tuning and reinforcement learning in the second phase to teach the model to use these memories actively and deliberately.
More from Models
- Claude 20x users report sharply tighter limits and faster quota burn — MarcJSchmidt · 2026-07-21
- Cola launches July, the latest model jokingly billed as “second only to Fable” — oran_ge · 2026-07-21
- Kimi K3 looks stronger and about 5× cheaper on a frontend dashboard task — OwariDa · 2026-07-21
- Last Week in AI recap: Anthropic’s $65B round, IPO filing, and Microsoft’s MAI push — Last Week in AI · 2026-07-21
- A user says Claude 4.6 felt worse yesterday and asks whether model quality can drift over time — Rahios · 2026-07-21
- Kimi K3 hits 89.4% peak on software tasks while Fable 5 is slightly steadier — FinanceYF5 · 2026-07-21