A neural net learns 2^100 LDPC codewords from just 10-20 examples
DimaKrotov · x · 2026-10-26
Dima Krotov shares research showing a Dense Associative Memory can learn the entire space of 2^n LDPC codewords from only 10-20 examples — roughly log(n) samples suffice — without ever being given the generation algorithm.
Key points:
- Grokking in LLMs and the memorization-to-generalization transition in diffusion models are related phenomena: the model recovers the true data-generating process from few samples.
- LDPC codeword generation is a precise mathematical process (solutions to parity-check equations), offering a controllable testbed where natural datasets fall short analytically.
- The accompanying chart shows an abrupt phase-transition-like jump in recovered codebook size around P≈16 training examples.
More from Research
- EMNLP 2026: sub-billion-parameter LMs show superhuman entity tracking, human-like fleeting memory — GretaTuckute · 2026-10-27
- 10-year study documents first 'unnatural gradient' phonotactic restriction in Tarma Quechua — begusgasper · 2026-10-27
- Clinical System One v0.2: top clinical AI drops from 97% to 65% on harder cases — MaziyarPanahi · 2026-10-27
- IdeaAnchor mines 14K papers to teach LLMs literature-grounded ideation — armancohan · 2026-10-26
- MIT's Ziming Liu: AutoResearch doesn't generalize — blind training tricks cause 'forking' disasters — ZimingLiu11 · 2026-10-26
- Stanford Data-Centric AI Workshop reopens site, extends poster deadline — jure · 2026-10-26