ICML Paper: Achieving Neural Network Sparsity via Rescaling Symmetries and Weight Decay
burny_tech · x · 2026-08-11
Highlights an ICML 2023 paper (arXiv:2210.01212) showing that adding artificial rescaling symmetries to neural networks makes them sparse. Because rescaling symmetry combined with weight decay is equivalent to L1 sparsity, this can be used to train highly sparse models. The proposed method, spred, acts as an exact differentiable solver for L1 penalties using standard SGD, demonstrating usefulness in gene selection and network compression.
More from Research
- AI Mechanistic Interpretability Resources: Key Articles & NeurIPS Workshop — burny_tech · 2026-08-11
- Training a 1B-Parameter LLM from Scratch for Just $200: Full Details Open-Sourced — SevereTilt · 2026-08-11
- SF DSPy Meetup Agenda Revealed: Focus on Flex and GEPA Optimization — dbreunig · 2026-08-11
- InfoWorld: Traditional Observability Shows 'What', AI Explains 'Why' — rseroter · 2026-08-11
- GPT-5.6 Solves Two Open Graph Theory Problems Unresolved for Decades — No-Performer-2242 · 2026-08-11
- DuplexGen: Scenario-Adaptive Dialogue Turn-Taking via Human Preference Calibration — illinois · 2026-08-11