Memory as an Energy Landscape---Hopfield
Nima Dehghani
cs.NE, cond-mat.dis-nn, nlin.AO, physics.bio-ph, q-bio.NC
2026-09-02
Hopfield memory as energy descent: 0.138 is the T=0 retrieval spinodal for unbiased Hebbian patterns; attention matches one modern update only when keys equal values.
Hopfield 1982 is often compressed into the phrase "associative memory" or the number 0.14N. Threshold units, Hebbian association, correlation memories, and recurrent binary nets were already on the table with McCulloch–Pitts, Hebb, Amari, Anderson, Kohonen, and Little. What the PNAS papers added was a physical synthesis: content-addressable recall became descent in phase space, a corrupted cue an initial condition, retrieval a trajectory, error correction a basin, and capacity a reorganization of the landscape with load.
This is Chapter 5 of Dehghani's NeuroAI textbook. It is not another algorithm survey. It derives the binary and graded Lyapunov functions in full, ties 0.138 back to the Amit–Gutfreund–Sompolinsky ensemble and success criterion, then follows the energy program through polynomial dense associative memory, exponential interactions, and the single update that coincides with scaled dot-product attention.
Binary spins si ∈ {−1,+1}, symmetric couplings, zero diagonal. Local field hi, zero-temperature asynchronous update si ← sgn hi. Energy E = −1/2 ∑ Jij si sj. A single-spin flip changes energy by ΔE = −(s′−s)hi, strictly downhill when the spin disagrees with a nonzero field. Finite state plus nonincreasing energy forces the asynchronous dynamics to a fixed point in finitely many steps. The proof uses three facts: symmetry, sequential updates, and arrival at a local minimum, which need not be the designed memory. Fully synchronous updates can still make two-cycles when J is symmetric.
Memories enter by the Hebbian outer product Jij = (1/N) ∑μ ξi^μ ξj^μ. Energy becomes a sum of squared overlaps, so lowering E rewards alignment with a stored pattern. Near a memory the local field splits into a retrieval signal plus crosstalk. A naive Gaussian estimate gives bit-error probability Φ(−1/√α). During retrieval the state correlates with the disorder, and mean-field theory dresses the noise variance as αr, with r renormalized by susceptibility.
The replica-symmetric equations collapse at T→0 to an error-function closure. Continuation in α kills the nonzero retrieval branch at αc ≃ 0.138. That number is the zero-temperature dynamical retrieval spinodal for dense, symmetric, Hebbian storage of unbiased random patterns in the thermodynamic limit. The 1984 circuit model replaces binary flips with monotone graded responses Vi=gi(ui). An inverse-response integral in the energy yields dE/dt = −∑ Ci g′(ui)(u̇i)^2 ≤ 0. Symmetry plus monotonicity suffice; binary states do not.
Later work sharpens the interaction F. Polynomial F(x)=x^n is dense associative memory: fixed bit-error capacity scales as N^{n−1}, whole-pattern zero error costs an extra ln N. Exponential interactions, under separation conditions, give K ∼ e^{cN} with 0<c<ln 2/2. A modern continuous Hopfield energy is log-sum-exp; the fixed-point update is x ← X softmax(β X^T x). Set V=K=X^T and β=1/√dk and this is algebraically scaled dot-product attention.
A fixed-seed 1982 replication script exposes mechanisms and does not replace proofs. With N=400, 18 random patterns, and a cue with 28% bits flipped, overlap climbs from 0.44 to 1 while energy falls; that demonstrates a basin, not a capacity theorem. Zero-temperature continuation terminates near α≈0.1376 at the plotted resolution. Finite-N synchronous runs round the transition, and mean overlap does not coincide with an m>0.9 criterion. The graded model with N=120, 4 patterns, and g=tanh(2.2u) drops energy to numerical precision and reaches target overlap about 0.966, with units still continuous.
Replica symmetry gives αc≃0.137905; one-step RSB moves it to 0.138186, two-step to 0.138187. Finite-size simulations are often summarized as near 0.14. A three-pattern majority-vote mixture has overlaps 1/2 each in the large-N limit and leading energy about −3N/8 against −N/2 for a pure memory, yet can be locally stable. Energy minima are not automatically designed memories.
| Shorthand | Precise claim |
| Capacity is 0.14N | 0.138 is the T=0 retrieval spinodal of that ensemble |
| Energy descent solves optimization | Descent reaches a stationary point, not a global optimum |
| Attention is a Hopfield net | One update coincides when keys and values are the stored patterns |
For people working on transformers and associative memory, the chapter puts slogans back under their assumptions. 0.138 is not a universal capacity; change the learning rule, sparsity, or success criterion and the number changes. Attention can equal one modern Hopfield update. Once layers, heads, residuals, causal masks, and separate Q/K/V enter, a transformer is not relaxing a symmetric energy to equilibrium. For neuroscience the effective-theory question is how much asymmetry, delay, adaptation, and ongoing drive can be integrated out before memory needs a nonequilibrium description.
The author's own boundary is sharp. Energy descent guarantees neither the intended memory nor a global optimum of an NP-hard cost. Real chemical synapses are directed; a generic recurrent circuit need not possess an energy. Delay, adaptation, spike timing, and drive can produce cycles, sequences, metastability, or chaos, all outside the proof. The Hebbian outer product is an embedding prescription, not a complete biological learning theory. Point neurons fail when dendritic nonlinearities or neuromodulation add state on the recall timescale. Exponential capacity needs norm and separation conditions; correlated patterns merge. The replication script inspects mechanisms: Figure 2 does not measure capacity, and Figure 6 does not prove exponential storage.