Hobbyist experiment compiles structured facts into frozen sparse memory for small models
sn2006gy · reddit · 2026-09-26
While building tiny-sparse-lab, a small open-source harness for testing N-gram/Engram-style conditional memory on models small enough for controlled experiments, the author stumbled onto an unplanned approach: instead of having a model learn an Engram table through LM training, compile external sparse memory directly from structured facts and train small models to consume it.
Bounded pilot on synthetic data:
- correct memory: 1.000
- incomplete memory: 0.5625
- random / disabled memory: 0.125
- conflicting memory: 0.000
A larger portability harness ran 120 controlled smoke arms (token- vs byte-addressed memory, structured semantic memory, two recipient widths, seeds 17/41/73, plus disabled/random/corrupted/frozen/adapter/native-memory controls). Artifact identity checks, recipient isolation, A→B→A replay, and retrieval traces all worked — but as deliberately two-update smoke tests, behavioral accuracy was 0 across the board, proving the machinery rather than portability.
This yields two research questions:
- Learned Engram portability — freeze a jointly-trained memory table, transplant it to differently-sized models with only a tiny recipient adapter, and test whether held-out entries survive.
- Compiled memory delegation (the more interesting one to the author): why make every model rediscover static structure via SGD, when Wikidata/WordNet/APIs/formulas can be compiled into external sparse memory, letting small models learn only language, routing, composition, and reasoning — i.e., how much static world structure needs to live in weights at all?
The author explicitly disclaims generality and that reasoning reduces to lookup, and lays out full control-group experiment designs for both directions.
More from Research
- Claude one-shots a nine-loop scattering amplitude, a frontier physics calculation — Dr_Singularity · 2026-09-26
- Scooped by a Machine: Claude Computes N=4 Super Yang-Mills to Nine Loops — KyleCranmer · 2026-09-26
- FrontierSmith accepted to NeurIPS as Spotlight, team says more work coming — AlexGDimakis · 2026-09-26
- RGBD20K: new RGB-D segmentation benchmark with 20K image pairs and 160 categories — UNT · 2026-09-26
- Formally verified Ethereum consensus proposal moves toward 4-8x faster finality — DavideCrapis · 2026-09-26
- Fewer than 10% of preregistered studies include multiple-hypothesis corrections — RexDouglass · 2026-09-26