Hilbert Operator for Progressive Encoding (HOPE): A Mathematical Framework for Deconstructing Learned Representations in Deep Networks
Hossein Mobahi, Peter L. Bartlett
cs.LG, cs.AI, stat.ML
2026-07-23
Google DeepMind models each neuron as a rank-1 Hilbert-Schmidt operator, analytically measuring functional capacity from batch normalization statistics alone to enable data-free progressive compression; the derived DEFT transfer framework scores H-Score 65.82 versus 12.54 for EWC.
Batch normalization creates a scaling symmetry that breaks magnitude-based pruning. Scale a neuron's incoming weights by 10 and BN divides by 10 during normalization — downstream output is unchanged, but the magnitude ranking is now wrong. L1-norm and BN-scale pruning both fall victim to this: a neuron ranked unimportant may simply have its weights compressed by BN, not be functionally weak. Activation-based alternatives avoid this but require forward passes through real data, expensive when data is unavailable or privacy-constrained.
HOPE lifts neurons from parameter space into Hilbert function space. Each neuron becomes a rank-1 Hilbert-Schmidt operator fi = gi ⊗ wout,i, where gi(x) = Ψ(wineff^T x + bi) is a continuous scalar activation landscape and wout,i is the output weight vector. Functional capacity is the Hilbert-Schmidt norm: ‖fi‖H = ‖wout,i‖2 × √K(i,i), where K(i,i) is the expected squared activation energy.
For data-free computation, HOPE applies the Maximum Entropy principle to BN moving statistics (μi, σi²), constructing a Gaussian surrogate PX and deriving a closed-form K(i,i) for ReLU. Zero forward passes are needed.
Three compression operations share the same cost functional: pruning projects a neuron to the null operator; merging finds the rank-1 approximation of the joint 2D subspace spanned by two neurons; block eviction collapses an entire residual branch, costed as an upper bound relative to skip-connection capacity. A greedy optimizer ranks candidates by distortion rate J/ΔP and executes the cheapest action each step.
DEFT (Dispersed Elastic Fine-Tuning) extends HOPE to transfer learning: a capacity threshold partitions the network into a frozen universal core (high-capacity neurons) and a plastic peripheral slack (low-capacity neurons). Connections from upstream slack to downstream core are severed at initialization, blocking target gradients from corrupting source representations.
Model compression on Keras ResNet-50 (ImageNet pretrained): HOPE's accuracy-vs-density curve consistently outperforms three structured baselines (L1-norm input pruning, L1-norm joint pruning, BN-scale pruning) at all density levels. Results are graphical only — no numerical table.
Cross-domain transfer (CIFAR-100 → SVHN, 4 trials × 5 tasks = 20 scenarios):
| Method | Target (SVHN) | Source (CIFAR) | H-Score |
| DEFT (Ours) | 89.79 ± 0.84 | 52.14 ± 5.29 | 65.82 ± 3.96 |
| Head-Only FT | 36.11 ± 2.79 | 63.13 ± 4.62 | 45.79 ± 2.05 |
| Full FT | 94.09 ± 0.28 | 7.52 ± 1.63 | 13.88 ± 2.84 |
| EWC | 93.94 ± 0.22 | 6.74 ± 1.74 | 12.54 ± 2.99 |
| PEFT | 81.91 ± 0.49 | 5.44 ± 0.98 | 10.18 ± 1.63 |
Full fine-tuning achieves the highest target accuracy (94.09%) but catastrophic forgetting reduces source retention to 7.52%. DEFT trails Full FT by 4 percentage points on SVHN while retaining 52.14% CIFAR accuracy, pushing H-Score from 13.88 to 65.82.
HOPE answers what structured compression is actually optimizing: functional capacity, not parameter magnitude. The framework is hyperparameter-free (no pruning ratio or importance threshold to tune) and data-free for BN networks. DEFT enables continual learning without source data, relevant for privacy-sensitive deployments, outperforming EWC by over 53 H-Score points while eliminating the O(N³) Fisher matrix computation EWC requires.
The paper explicitly labels these proof-of-concept experiments. Compression covers a single ResNet-50 checkpoint; transfer uses a small 8-layer VGG-style network on CIFAR-100/SVHN. No comparison against LoRA or DoRA. For architectures using LayerNorm or RMSNorm instead of BN, a calibration forward pass over a small data batch is needed. The zero-bias approximation for cross-kernels introduces systematic error in layers with large bias terms.