Google DeepMind paper says raw weight size can mislead neural network analysis
burny_tech · x · 2026-07-28
Google DeepMind’s HOPE paper argues that judging neural networks by raw weight magnitudes is misleading because scale symmetries can make large weights look important and small weights hide key features.
- The paper frames model compression as a way to study learned representations.
- It introduces Hilbert Operator for Progressive Encoding (HOPE), a data-free and hyperparameter-free framework.
- HOPE models neurons as rank-1 Hilbert-Schmidt operators and unifies pruning and neuron merging as low-rank subspace projection.
- The authors also extend the idea to macro block eviction so larger residual structures can be treated under a single metric.
- They present proof-of-concept experiments in compression and fine-tuning to show the approach’s practical potential.
More from Research
- Netflix details its production LLM judge: hundreds of thousands of recommendations scored weekly — omarsar0 · 2026-08-24
- Nature Comment: Provenance, not interpretability, grounds trust in autonomous science — gabepgomes · 2026-08-24
- New Architecture RHEA: Train 1B Model on 8GB VRAM — zemondza · 2026-08-24
- Trained two 16M-param models to do generative CAD with real physics — debreuil · 2026-08-24
- Claude model helps discover complex structure on S^6, solving 60-year-old math problem — Singularitarian · 2026-08-24
- Study: Agents read instructions/notes 60.5% of the time, rarely touch API docs — dair_ai · 2026-08-24