Four neural network myths debunked: depth, loss, ReLU, and Adam vs SGD
cneuralnetwork · x · 2026-10-31
cneuralnetwork debunks common neural network myths: deeper isn't always better (optimization and generalization still matter), low training loss can mean memorization, ReLU doesn't fully solve vanishing gradients (dead neurons persist), and Adam isn't always better than SGD (which can generalize better). He invites others to add more myths.
More from Research
- KAN boosts TabPFN scaling forecasts: OOD error cut up to 80% on StepLaw tasks — ZimingLiu11 · 2026-10-31
- Pre-registered training-curve forecast hits: TabPFN covers 14/14 later checkpoints — ZimingLiu11 · 2026-10-31
- KAN author argues training-curve prediction fits tabular foundation models like TabPFN, not LLMs — ZimingLiu11 · 2026-10-31
- Code-switched questions hurt LLM accuracy; CANVAS steering fixes internal representations — burkov · 2026-10-31
- KV Cache explained with chai: why token 500 doesn't redo tokens 1-499 — cneuralnetwork · 2026-10-31
- Qwen Image 2.1 2-step distill LoRA hits v1.5, enabling realtime editing on a single GPU — linoy_tsaban · 2026-10-31