ReLU networks have zero second derivative almost everywhere — why PINNs need smooth activations

bravo_abad · x · 2026-11-22

The author explains why twice-differentiating a ReLU network yields zero almost everywhere: ReLU's derivative is 0 or 1, making the network piecewise linear with piecewise-constant first derivative and vanishing second derivative.

Takeaway: backprop needs one derivative of the activation; the model you want may need more.

Original post →

More from Research

Research channel →