On a generalization of the Jensen-Shannon divergence and the JS-symmetrization of distances relying on abstract means
Frank Nielsen
cs.IT, cs.LG
2019-04-08
Replacing the arithmetic mixture in Jensen-Shannon with a geometric mean yields closed-form JS between exponential-family densities, including multivariate Gaussians.
Jensen-Shannon (JS) is the usual way to symmetrize Kullback-Leibler divergence and cap it at log 2: average the two KL numbers to the arithmetic mixture (p+q)/2. The square root is a metric. The divergence shows up in GANs, clustering, and quantum information.
On parametric families the mixture is the obstacle. The arithmetic blend of two Gaussians is not Gaussian, and the same holds for a generic exponential family. Textbook JS therefore has no closed form between Gaussians, so implementations fall back to quadrature. Jeffreys divergence symmetrizes KL without an upper bound and is numerically noisier.
Drop the insistence on the arithmetic mean. Any abstract mean M that sits between its two arguments defines a pointwise mixture; after normalization this is an M-mixture. Plug that mixture into the JS template and you get M-JS. The same template JS-symmetrizes an arbitrary base distance D, optionally with a second mean N on the two directed terms.
The mean is chosen so the mixture stays in-family.
Ordinary JS is the special case M = arithmetic mean. G-JS is a different divergence. Boundedness of M-JS is guaranteed when M dominates the arithmetic mean; geometric and harmonic means do not, so G-JS need not inherit the log 2 cap.
The paper is a closed-form paper. There is no SOTA table on a learning benchmark. The numbers that can be checked sit in the formulas.
| Object | Distance | Result | ||||||
| Ordinary JS on any pair | Upper bound | 0 to log 2 | ||||||
| Scale Cauchy, γ=0.1 vs 0.5 | Harmonic JS | about 0.176 | ||||||
| Scale Cauchy, γ=0.2 vs 0.8 | Harmonic JS | about 0.129 | ||||||
| Same exponential family | Dual geometric JS | skew Jensen divergence of F | ||||||
| Multivariate Gaussians | Dual geometric JS | ½[(1-α)μ1ᵀΣ1⁻¹μ1+αμ2ᵀΣ2⁻¹μ2-μαᵀΣα⁻¹μα+log(\ | Σ1\ | ^{1-α}\ | Σ2\ | ^α/\ | Σα\ | )] |
Here Σα=((1-α)Σ1⁻¹+αΣ2⁻¹)⁻¹ and μα=Σα((1-α)Σ1⁻¹μ1+αΣ2⁻¹μ2). Skew Bhattacharyya distance on an exponential family is exactly that Jensen divergence; G-JS folds the identity into one symmetrization.
A clustering section hooks G-JS to a generalized k-means++ guarantee via a quasi-triangle inequality. No clustering accuracy on real data is reported.
If you need a symmetric divergence between Gaussians, exponential-family densities, or scale Cauchys, the previous options were quadrature or a switch to Jeffreys or Wasserstein. The geometric-mean route is differentiable and can sit inside a k-means update. The cost is that the object is no longer classical JS. Later GAN work used geometric JS; this paper is the one that makes the "change the mean, keep the family" rule explicit.
Everyday discrete JS in NLP eval does not need this. Information geometry and probabilistic clustering do.
Closed forms exist only when the mean keeps the mixture in-family. Ordinary JS between Gaussians is still without a closed form; the problem is bypassed. Upper bounds, the metric property, and f-divergence status do not transfer automatically from classical JS. Matrix JS, including quantum JS, is defined rather than evaluated. There is no large-scale learning experiment and no head-to-head with numerical JS on a downstream task. A later addendum expands properties; practical value depends on whether you actually want a closed symmetric substitute for KL on Gaussians.