Reddit thread: softmax has only N-1 degrees of freedom — drop one input?
Kinexity · reddit · 2026-09-29
A Reddit user proposes that since softmax outputs are constrained to sum to 1, they have only N-1 degrees of freedom, so the layer before it could take N-1 inputs by assuming logits sum to zero, deriving the last logit as minus the sum of the others. This would trim parameters from the final layer and might speed convergence slightly, though the poster concedes the benefit is negligible in almost all cases and asks if there are theoretical reasons against it.
More from Research
- TT-VidT: temporal-decoupled video pretraining accepted at NeurIPS — CMHungSteven · 2026-09-29
- EgoDemo, an egocentric human demonstration dataset for embodied AI, trends on HF — LightwheelAI · 2026-09-29
- Peking University: post-training leaves behavioral shadows, one word per prompt transfers coding skill — PekingUniversity · 2026-09-29
- RenderRank reranks documents as images: 35% fewer tokens, beats sub-4B text rerankers — nlpai-lab · 2026-09-29
- DepthBench: residual connection design decides whether depth is a real scaling axis — Intelligent-Systems · 2026-09-29
- DN-MOPD: domain-normalized feedback fixes multi-teacher on-policy distillation imbalance — NanyangTechnologicalUniversity · 2026-09-29