Google DeepMind paper says raw weight size can mislead neural network analysis
burny_tech · x · 2026-07-28
Google DeepMind’s HOPE paper argues that judging neural networks by raw weight magnitudes is misleading because scale symmetries can make large weights look important and small weights hide key features.
- The paper frames model compression as a way to study learned representations.
- It introduces Hilbert Operator for Progressive Encoding (HOPE), a data-free and hyperparameter-free framework.
- HOPE models neurons as rank-1 Hilbert-Schmidt operators and unifies pruning and neuron merging as low-rank subspace projection.
- The authors also extend the idea to macro block eviction so larger residual structures can be treated under a single metric.
- They present proof-of-concept experiments in compression and fine-tuning to show the approach’s practical potential.
More from Research
- MIT's Neil Thompson: AI comes for tasks before jobs, and fixed costs favor big firms — bigdata · 2026-10-16
- UMD's PAIR framework bridges perception and action in VLA models via bridge tokens — Kaixi Feng · 2026-10-16
- CI-MSE offline metric hits -0.87 Spearman correlation with real robot task success vs weak raw MSE — chris_j_paxton · 2026-10-16
- Colossal Biosciences uses AI to decode genetic switches behind the extinct dodo — castrotech · 2026-10-16
- Colossal Biosciences uses AI to identify gene switches in dodo de-extinction push — castrotech · 2026-10-16
- Cheng Lou open-sources Handwired: hand-written weights turn code into training-free neural nets — dotey · 2026-10-16