Neural Spectral Capacity: Measuring and Designing Architectures from Network Specification Alone
Chenyu Zhu, Ruoyu Zhao, Zhichao Lu
cs.LG, cs.AI
2026-09-19
CityU scores nets from matrix shapes via Marchenko-Pastur. FlexiBERT τ=0.695; DP finds Transformer-XL at PPL 23.087 in 2s; LLaMA-7B prunes to 5.7B at 65.47 Avg8, no calibration.
Designing or compressing a Transformer is a capacity-allocation problem: depth versus width, FFN ratio, heads, experts. #Params and #FLOPs see size and compute, not structure. Two nets with the same parameter budget and different depth-width or head layouts get the same score and then train differently. Zero-cost proxies such as SNIP, W-PCA, and ZeroLM can rank, but each candidate must be instantiated, fed inputs, and scored from activations or gradients. The number moves with the seed and the batch. They are black boxes, so search is evolutionary: you get the best architecture visited, not the best architecture under the proxy.
What is missing is a scalar that reads the spec, cares about structure, and can be optimized exactly.
NSC treats each weight matrix as a linear Gaussian channel. Spectral capacity is ln det(I + WᵀW), the sum of ln(1+σᵢ²) over singular values, Telatar mutual information without the 1/2. Under standard random init, squared singular values follow the Marchenko-Pastur law, so expected capacity collapses to a closed form ψMP of the shape (m, n) and init variance s². Fix Xavier or Kaiming and s is a function of shape; the network score is the sum of per-matrix ψMP.
Counted matrices: per-head Q/K/V, the output projection, and FFN weights. Convolutions are reshaped to the im2col map. Embeddings, biases, norms, and nonlinearities are left out. Heads are scored separately so head count is visible. Within a layer the sum is the parallel-channel rule; across layers it is an empirical choice, and ablations prefer it over a min bottleneck.
Because both the score and typical resource costs add over layers, maximizing NSC is a knapsack. NSC-DP enumerates network-level choices (depth, embedding size) and solves per-layer heads and FFN widths by exact DP. One run also yields the Pareto front over budgets. A ψMP evaluation is about 10 μs; spaces up to 10³² finish in seconds on one CPU.
On 500 FlexiBERT variants, Kendall τ is 0.695 for NSC, 0.485 for #Params, 0.552 for #FLOPs, 0.635 for W-PCA. Restrict to pairs within 10% parameter count and #Params falls to 0.082 while NSC holds 0.505. Scoring all 500 takes 2 ms on CPU; W-PCA takes 88 s on an A100. The 10%-PW lead over #Params repeats on AutoFormer-Tiny, NATS-Bench-SSS, and MobileNetV3.
Transformer-XL, 38.4M non-embedding params, trained from scratch on WikiText-103:
| Method | Test PPL | Search |
| Transformer-XL Base | 23.279 | — |
| Synaptic Diversity | 23.135 | 903 s |
| Softmax Confidence | 23.532 | 883 s |
| W-PCA | 23.669 | 965 s |
| NSC-DP | 23.087 | 2.0 s |
AutoFormer-Tiny at 5.8M: NSC-DP Acc@1 75.276 versus the 24-GPU-day oracle at 75.308, search 0.03 s.
Structured pruning of LLaMA-7B to 5.7B on the LoNAS supernet, no calibration data. Commonsense Avg8: NSC-DP 65.47, W-PCA 63.83, GradNorm 62.45. NSC-DP wins every task, 0.5 s on CPU versus 45 min on GPU for W-PCA, about 5900×.
Architecture search and structured pruning get a spec-only objective that needs no instantiation, data, or gradients, and that DP can solve exactly. The WikiText-103 PPL drop from 23.279 to 23.087 is small. The pruning result is the one that changes the workflow: NSC never looks at pretrained weights or a calibration batch, unlike Wanda, SparseGPT, or LLM-Pruner. Use it to screen structure, then spend GPU time on real training. An appendix also shows that uniform per-layer FFN width is optimal for this proxy on Transformer-XL, so DP is not quietly inventing lopsided layer widths.
It remains a proxy. DP maximises NSC, not the downstream loss. Summing sequential layers has no information-theoretic reading; it just won the aggregation ablation.
There is no limitations section. Visible gaps: the LLaMA study only varies FFN width and LoRA rank inside LoNAS, and the chosen subnet still inherits supernet weights. At init the whole spectrum sits in the MP bulk; trained heavy tails are out of scope by design. Perfect init-to-PPL correlation on 10 GPT-2 nets is a small, size-spread sample; the large-scale range is Spearman ρ 0.78-0.97. A 0.2 PPL gain is modest. The channel model defines ψ; it does not claim the forward pass is that Gaussian channel.