Why L1 performs feature selection and L2 doesn't: the geometry of regularization
techNmak · x · 2026-11-03
Two models can share the same dataset, features and loss, yet one eliminates several features while the other keeps them all — the difference can boil down to choosing L1 vs L2 regularization.
Author techNmak's explainer:
- Penalty shape: L1 penalizes the sum of absolute coefficients, L2 the sum of squares. L2 shrinks coefficients without zeroing them; L1 can drive coefficients exactly to zero, effectively removing features.
- Geometry: in 2D, the L1 constraint region is a diamond while L2's is a circle. Minimizing convex loss under these constraints, the diamond's corners make axis-aligned solutions likely — and an axis solution means a coefficient is exactly zero. That's why Lasso does feature selection during training while Ridge keeps nonzero coefficients.
- Trade-offs: with strongly correlated predictors, Lasso may pick one and suppress another; Ridge spreads coefficients more smoothly. Neither is universally better — scaling, regularization strength and data structure all matter.
Core insight: regularization doesn't just control weight size, it shapes which features the model uses at all.
More from Research
- 6th Applied Active Inference Symposium opens registration: free, online, Karl Friston keynotes — AnnaCiaunica · 2026-11-03
- AutoRLBench: A Benchmark Testing AI Agents on Real RL Training Failures — wzenus · 2026-11-03
- GPT-6 Astra edits 0.016% of Qwen3.5-0.8B weights, boosting math up to 6x with no training — _lewtun · 2026-11-03
- Maybe AI excels at Navier-Stokes because fluid flows have no GDPR, researcher quips — lescarr · 2026-11-03
- Yoav Goldberg: If you can write a Python program for a task, it's not worth RL-ing over — yoavgo · 2026-11-03
- Anthropic open-sources FlashProve: provable inference costs ~165x more than raw inference — arthurcolle · 2026-11-03