ReLU networks have zero second derivative almost everywhere — why PINNs need smooth activations
bravo_abad · x · 2026-11-22
The author explains why twice-differentiating a ReLU network yields zero almost everywhere: ReLU's derivative is 0 or 1, making the network piecewise linear with piecewise-constant first derivative and vanishing second derivative.
- Training is unaffected: backprop only needs a first derivative; the activation's job is just preventing collapse into a linear map (without nonlinearities, a 269,322-parameter network equals a 7,850-parameter linear model).
- Derivative-based use cases break: forces from a ReLU potential energy surface jump no matter how accurate the energies look; a physics-informed network on the heat equation loses its diffusion term before training starts.
- Fix: smooth activations like Softplus, SiLU and GELU round off the corner while keeping ReLU's shape, with nonzero derivatives at every order.
Takeaway: backprop needs one derivative of the activation; the model you want may need more.
More from Research
- w2v-bert-fast open-sources exact 3x speedup for w2v-BERT 2.0 training and inference — kastnerkyle · 2026-11-23
- AI reads 2,000 historical illustrations: individual errors, near-perfect aggregate patterns — _akpiper · 2026-11-23
- Princeton/Meta evolve inference strategies: 39% cheaper Qwen3 runs, same accuracy — michaelTM_ai · 2026-11-23
- MIT approach enables physically consistent AI models of chaotic systems — SnoozeDoggyDog · 2026-11-23
- Sinclair lab uses AI to find chemical age-reversal molecules, mouse tissues rejuvenated in 4 weeks — thederbiedone · 2026-11-22
- Aaronson on reported LLM proof of the Riemann Hypothesis: 17.4M lines of Lean code — jedisct1 · 2026-11-22