The Spectral Neuron
Alex Shtoff
stat.ML, cs.LG
2026-08-08
Spectral neuron: read the k-th eigenvalue of an affine matrix pencil. On HIGGS, d=11 hits log loss ~0.52 vs linear ~0.64.
Tabular work often wants two things at once: enough accuracy, and coefficients you can show a person. An insurer has to explain a risk score. An ad bidder needs P(win) to rise with bid. A regulator wants to know which feature moved the number. Linear models put the story in the weights and then run out of expressivity. Trees and nets fit more, but a tree can jump by an arbitrary amount, and the map from input to output is no longer something you read off a coefficient.
Shape-constrained nets and lattices can force monotonicity or convexity, at the cost of a special architecture. GAMs and NAMs stay readable by restricting interactions. The missing object is a primitive that gets more expressive as it grows, keeps shape constraints by construction, and still exposes feature influence from its parameters.
A spectral neuron replaces the scalar weights of a classical unit with symmetric matrices, and replaces the pointwise nonlinearity with an eigenvalue. For \(x \in \mathbb{R}^n\) it learns \(d \times d\) real symmetric matrices \(A0,\ldots,An\) and predicts
\[fk(x)=\lambdak\bigl(A0+\sumi xi Ai\bigr)\]
where \(\lambdak\) is the \(k\)-th smallest eigenvalue. Matrix size \(d\) is the scaling knob. The index \(k\) is the shape knob. Alex Shtoff at the Technology Innovation Institute treats this as a trainable machine-learning primitive, not a new optimizer.
Because the nonlinearity is an ordered eigenvalue of an affine pencil, classical spectral facts become model properties:
Training uses off-the-shelf autograd. PyTorch's torch.linalg.eigh already backprops through a Clarke subdifferential of \(\lambdak\). Initialization has to avoid two traps. If the matrices are simultaneously diagonalizable, the model collapses to an order statistic of affine functions (maxout at the top eigenvalue), and updates that are linear combinations of past gradients stay inside that set. If the eigengap is tiny, Davis-Kahan says the eigenvector, hence the gradient, is hypersensitive. The recipe puts \(A0\) at \(Q^\top\mathrm{diag}(-1,\ldots,0,\ldots,1)Q\) with the zero in position \(k\), and sets each \(Ai\) to \(\alphai I\) plus a small diagonal jitter, which keeps the gap at least \(1/2\) at init for typical standardized features.
The compute is the trade. A linear model costs about \(2n\) FLOPS. Training here is about \(n d^2+(14/3)d^3\), inference \(n d^2+(4/3)d^3\). Expressivity, shape control, and coefficient transparency sit on the same matrices.
The paper does not claim SOTA. Experiments are scaling curves, Adam only, mid-eigenvalue, odd \(d\) so the middle index is unique. Numbers below are read from the figures; the text does not tabulate them.
On univariate synthetic targets, harder functions need larger \(d\). Noiseless complexity-10: \(d=5\) RMSE flattens near 0.2, \(d=15\) keeps falling to about 0.015. Complexity-20: \(d=5\) near 0.55, \(d=15\) near 0.09. When the target is monotone, the PSD constraint helps at small sample counts. A bivariate target of complexity 13 keeps improving at \(d=15\); smaller \(d\) flatten as more data arrives.
Two real datasets, both cross-entropy, batch 4096.
| setting | data | baseline | spectral neuron |
| HIGGS, \(d=3\) (174 params) | 11M rows, 28 numeric, up to \(2^{26}\) samples | linear 0.64 | log loss 0.57 |
| HIGGS, \(d=11\) (1914 params) | same | linear still 0.64; 2- and 3-layer MLP 0.505 | 0.52 |
| Criteo, \(d=3/7/11\) | 45.8M rows, 26 cat + 13 num, up to \(2^{28}\) | linear and FM cluster at 0.45–0.46 | continuous-feature spectral slightly lower, 0.45 |
On Criteo, treating numeric columns as numbers (\(\ln^2(1+x)\) then standardize) beats binning. Factorization machines that match parameter count do not improve as embedding dim goes from 5 to 65, and the dim-65 run rises to about 0.475 at \(2^{28}\). Spectral neurons improve modestly with \(d\).
On HIGGS, a one-feature corruption \(\delta=\varepsilon\sigmai\) with \(\varepsilon\in[-0.5,0.5]\) gives \(|\Delta f|/(|\delta|\|Ai\|2)\) well below 1 for most features, so the bound is not vacuous. As \(d\) grows the ratios pile up near 0: the bound loosens, which is why the paper flags spectral-norm regularization as future work.
This is a modeling primitive, not a leaderboard architecture. It is for tabular jobs that already know some coordinates must be monotone or convex, and that want feature influence written in the same currency as a linear coefficient. A hypernetwork can emit the matrices from unconstrained context features, while the spectral neuron enforces a CDF-shaped bid response. That is cleaner than bolting constraints onto a deep net.
Depth-2+ MLPs still win slightly on HIGGS. If log loss is the only score, there is no reason to switch. The reason to switch is that the same parameters give you shape guarantees and influence bounds, and growing \(d\) does not retire the explanation channel.
Symmetric eigensolves are \(O(d^3)\). There is no custom kernel for a single eigenpair. Experiments only used the middle eigenvalue, so the learnability of the convex and concave endpoints is not tested at the same scale. Multiple PSD matrices were barely explored; monotonicity in practice was "the last feature, diagonal and nonnegative".
Real data is Criteo and HIGGS only. The global influence bound loosens with \(d\), so coefficient transparency thins out on large matrices. The construction already lived in the PMM framework; the contribution is treating it as a trainable ML primitive with an init recipe, shape control, and scaling plots. The init heuristic assumes roughly standardized features that rarely exceed 5 in magnitude. A skewed distribution can land in a small gap.