q-Neurons: Neuron Activations based on Stochastic Jackson's Derivative Operators
Frank Nielsen, Ke Sun
cs.NE, cs.LG, stat.ML
2018-06-01
Wrap any activation in a stochastic Jackson q-derivative (one extra forward call). Curved functions hit 99.65% on MNIST CNN and 82.9% q-ELU on CIFAR-10; ReLU barely moves.
Stock neuron activations are deterministic. ReLU, tanh, ELU, and softplus return the same output for the same input. Randomness in training usually comes from minibatches, dropout, or noise added to gradients. Frank Nielsen at Sony CSL and Ke Sun at Data61 take Jackson's q-derivative, a 1909 finite-difference operator, and turn it into a stochastic activation that is noisy in the forward pass and carries second-order information in its gradient.
Ordinary derivatives need a limit. Jackson's q-derivative skips the limit and uses a finite difference: the gap between f(x) and f(qx), divided by (1-q)x. As q approaches 1, it recovers the usual derivative. Machine learning had barely touched this tool; the closest prior work used q-gradients for global optimization.
For any activation f, the q-version is
gq(x) = [f(x) - f(qx)] / (1-q)
which is Jackson's q-derivative multiplied back by x. q is a random variable with mean 1. As the variance of q goes to 0, gq(x) converges to f'(x)·x, not to f(x). The derivative of gq itself goes to f'(x) + f''(x)·x.
That is the actual change: the shape of the activation. In the small-noise limit the forward value looks like "derivative times input", and the backward gradient picks up a second-order term. Each neuron draws a chord between f(x) and f(qx) and uses the chord slope in place of the tangent.
Sampling in the experiments: draw ε from a standard Gaussian, then set q = 1 ± (λ|ε| + φ), with the sign of ε. φ is fixed at 10^{-3} to avoid division by zero. The only hyperparameter is the scale λ. It can sit at 0.02 or 0.1 for the whole run, or anneal as λ = λ0 / (1 + γ(T-1)). They train 100 epochs with γ=0.5, so λ near the end is about 0.02 λ0.
The Python core is three lines: sample a Gaussian with the same shape as x, evaluate the base activation twice, take a difference. No extra trainable weights. Cost versus the original activation is one extra forward evaluation plus one sample. Both q-activations and dropout change the forward pass; the experiments run both with and without dropout.
Piecewise-linear ReLU has second derivative zero almost everywhere, so gq nearly coincides with f. Curved activations (ELU, tanh, softplus) are the intended targets.
MNIST and CIFAR-10, a shallow CNN and a two-layer MLP, SGD without momentum, 100 epochs. MNIST uses batch 64 and learning rate 0.05; CIFAR uses batch 32 and learning rate 0.01. Each curve averages 10 independent runs. Baselines are vanilla tanh, ReLU, ELU, and softplus.
| Setup | Vanilla test acc. | Best q-version |
| MNIST CNN, no dropout, tanh | 99.57% | qtanh annealed λ0=5, 99.65% |
| MNIST CNN, dropout, ELU | 99.62% | qelu fixed λ=0.1 or annealed λ0=5, 99.65% |
| MNIST MLP, no dropout, ELU | 99.04% | qelu annealed λ0=5 or 9, 99.17% |
| CIFAR CNN, no dropout, tanh | 78.76% | qtanh annealed λ0=5, 80.63% |
| CIFAR CNN, dropout, ELU | 81.91% | qelu annealed λ0=1, 82.94% |
| CIFAR CNN, dropout, softplus | 70.64% | qsoftplus fixed λ=0.1, 80.14% |
Best MNIST CNN error is 0.35%, from q-ELU and q-tanh. Best CIFAR-10 is 82.9%, from q-ELU with dropout.
ReLU barely moves. On CIFAR without dropout the vanilla net hits 80.43%; the best q-ReLU is 80.44%. The paper's advice is to use q-neurons only with curved activations.
λ is not fragile. Fixing it at 0.02 or 0.1 already beats the vanilla activation in most plots; annealing squeezes a bit more. Large λ underfits. Gains show up with and without dropout.
The largest jump is CIFAR softplus: 72.02% to 78.40% without dropout, 70.64% to 80.14% with dropout. Vanilla softplus was already weak on this shallow CNN. The q-version pulls a weak activation back into a usable range; it does not set a record on a strong baseline.
In engineering terms this is a drop-in activation swap: three lines of code and one scalar λ. The 2018 write-up is bullish ("almost always replace the deterministic version"). Read now, it looks like a regularizer that ties second-order information to forward noise, in the same family as dropout and gradient noise, acting on the activation shape.
For small models still using ELU, tanh, or softplus and trained from scratch, λ=0.02 is cheap to try. The cost is a second activation evaluation. Modern stacks on GELU, Swish, or SiLU get no evidence from this paper. Skip it for ReLU.
The improvement is incremental. The authors say they are not chasing SOTA; they want the effect size and hyperparameter sensitivity. 82.9% on CIFAR-10 is a reasonable shallow-CNN number for 2018, not a record.
The study stops at MNIST and CIFAR-10, shallow CNNs, and a two-layer MLP. The optimizer is SGD without momentum. No ImageNet, no ResNet. MNIST gains are 0.05 to 0.1 percentage points, averaged over 10 runs, with no variance reported.
q-derivatives have no generic chain rule, so backpropagation is not replaced by q-calculus. gq is handed to ordinary automatic differentiation. The q in the name comes from q-calculus, not quantum computing; "quantum version" is a historical label.
Compute cost is a doubled activation call, which may not be free in deep nets. Underfitting from large λ is visible on the curves and is not analyzed as a failure mode. The appendix samples q as an offset from 1, while equation (6) samples Jackson's q directly; the two parameterizations differ by 1.
After 2018 the default activations moved to GELU and Swish. The "almost always replace" claim was never checked on those functions.