Fractale-350M: Open-Sourcing Memory as a Training Behavior

KKuettes · reddit · 2026-07-20

Independent researchers have released the Fractale-350M-base model, exploring a novel long-term memory mechanism by treating memory as a training behavior instead of relying on long contexts.

Core Design

Memory Bank Mechanism: The model's only long-term memory is a bank of 8 vectors. It writes a summary vector every 512 tokens and uses FIFO (First-In-First-Out) to discard the oldest memory.
Hypernetwork Reading: Vectors in the memory bank are expanded into low-rank MLPs via a hypernetwork, participating directly in the forward pass, bypassing traditional text retrieval or attention mechanisms.
Pre-training Objective: Forces the model to rely solely on these 8 memory vectors to predict the beginning of the next unseen text chunk.

Experimental Data

Memory Dependence (GAP): On code datasets, the cross-entropy difference between retaining and resetting memory reaches up to 9.4 nats (CE drops from 12.9 to 3.45); for web data, it is 7.3 nats. This indicates the model's deepening reliance on its own memory during training.
Mechanism Validation: Smaller-scale controlled experiments confirmed that memory content can be addressed via cues, survives after 2000+ steps, and transmits information bidirectionally (e.g., between docstrings and code).

Open Source & Limitations

This is a base model; it is not instruction-tuned or aligned, and its raw fluency aligns with expectations for 386M parameters (trained on 10 billion tokens).
The model only memorizes "key points" (like domains, structures, facts) and cannot recite source text verbatim.
Fully Open-Sourced: Includes model, code, complete research logs, and paper. Total pre-training cost was roughly $320 (8xA100 rental).

The author plans to use instruction tuning and reinforcement learning in the second phase to teach the model to use these memories actively and deliberately.

Original post →

Fractale-350M: Open-Sourcing Memory as a Training Behavior

More from Models