Neural Spectral Capacity: a training-free architecture score that beats #Params and prunes LLaMA-7B 5900x faster

CityU-HongKong · hf · 2026-09-25

Researchers from City University of Hong Kong propose Neural Spectral Capacity (NSC), a closed-form scalar based on the singular-value spectrum of weight matrices that measures and designs Transformer architectures without any training.

Core idea: #Params and #FLOPs capture size and compute but not structure—two architectures with identical parameter budgets but different depth-width, head, or FFN allocations score identically yet behave differently. Under standard random initialization, the Marchenko-Pastur law lets NSC be computed from the architectural specification alone, with no model instantiation, data, or gradients.

Key results:

NSC's layer-wise additive structure enables NSC-DP, an exact dynamic-programming solver that finds the globally NSC-maximizing architecture under resource constraints in seconds on a CPU—a guarantee black-box search over existing training-free proxies cannot provide.
It outperforms #Params, #FLOPs, and representative training-free proxies across seven Transformer and CNN families (τ=0.505 on FlexiBERT pairs differing by <10% params, where #Params collapses to 0.082).
NSC-DP discovers a Transformer-XL architecture on WikiText-103 beating the human-designed baseline in 2 seconds.
It prunes LLaMA-7B to the best 5.7B model across eight commonsense reasoning tasks with no calibration data, 5900x faster than the strongest training-free proxy baseline.

Original post →

Neural Spectral Capacity: a training-free architecture score that beats #Params and prunes LLaMA-7B 5900x faster

More from Research