Geometric Model Shows Fisher Information Resolves DNNs' Apparent Violation of Occam's Razor
FrnkNlsn · x · 2026-11-01
Frank Nielsen highlights his paper with Ke Sun, A Geometric Modeling of Occam's Razor in Deep Learning (arXiv:1905.11027, first posted 2019, revised v9 in 2025 and published in Information Geometry).
The paper tackles a puzzle: deep neural networks have enormous parameter spaces, seemingly defying Occam's razor, yet generalize remarkably well — something classical model-selection theory for regular models can't explain.
- Grounded in minimum description length (MDL) theory, linking simplicity to better generalization;
- Introduces locally varying dimensionality of the parameter space via the number of significant dimensions of the Fisher information matrix;
- Models the parameter space as a manifold using singular semi-Riemannian geometry, deriving complexity measures that yield short description lengths.
Key insight: Fisher information reveals DNNs' effective complexity is low — huge parameter counts are nearly degenerate (tangent vectors along null curves don't change the model), so high-dimensional parameter spaces don't actually violate Occam's razor, explaining DNNs' strong practical performance.
More from Research
- NeurIPS 2026 acceptance analysis ranks South Korea third by first-author affiliation — JungWooHa2 · 2026-11-01
- Semantic bandit experiments show irrelevant wording skews LLM exploration-exploitation balance — burkov · 2026-11-01
- RoboColiseum robot benchmark: Claude Opus 5.5 scores 76.28, edging out GPT-6 Astra — jiqizhixin · 2026-11-01
- ETH Zurich releases EgoHTR: 150k-frame egocentric 4D dataset for humanoid terrain traversal — Scobleizer · 2026-11-01
- DiffusionART-Flash open-sourced: RL for diffusion models with 25x fewer training timesteps — YangYou1991 · 2026-11-01
- NVIDIA's World from Motion accepted to NeurIPS 2026: dynamic 3D from monocular video — Scobleizer · 2026-11-01