Two tales for a geometric Jensen--Shannon divergence
Frank Nielsen
cs.IT, cs.LG
2025-08-07
Nielsen defines an unnormalized geometric JSD that is an f-divergence. It differs from the usual G-JSD by Z-log Z-1; neither square root is a metric, and both read as regularized ordinary JSD.
Jensen-Shannon divergence (JSD) is a symmetrization of KL. It is bounded by log 2, and its square root is a metric. It is widely used in machine learning, yet it has no closed form between Gaussians, because the differential entropy of a two-component Gaussian mixture is not analytic.
Geometric JSD (G-JSD) replaces the arithmetic mixture by a normalized geometric mixture. That version is closed-form for Gaussians, which is why it shows up in VAEs and perception-error models. Normalization generally destroys separability and the information monotonicity of f-divergences. Nielsen asks what happens if the geometric mixture is left unnormalized and KL is replaced by the extended KL that stays nonnegative on positive measures.
Two definitions sit side by side.
Both rewrite in terms of Jeffreys divergence J and Bhattacharyya distance B (or coefficient BC):
G-JSD = (1/4)J − B
extended G-JSD = (1/4)J + BC − 1
The gap is exactly Δ = Z − log Z − 1, where Z=∫√(p1 p2) is the geometric-mixture partition. On an exponential family that gap is the Jensen gap of the cumulant function. For Gaussians Z≤1, so the extended version is at most the usual one.
The extended G-JSD is the f-divergence for f(u)=(1/4)(u−1)log u + √u − 1, hence separable, information-monotone, and invariant in the information-geometry sense. Usual G-JSD is not separable in general, because of the mixture normalizer.
Both formulas also read as ordinary JSD plus a KL between the arithmetic mixture and the M-mixture, plus a partition correction when needed, i.e. as regularizations of JSD. The power-mean limits give min/max-JSD; extended max-JSD is at most total variation.
Gaussian formulas are assembled from the known Jeffreys and Bhattacharyya expressions, with Σ̄=(Σ1+Σ2)/2. Estimation is discussed via Monte Carlo and via projective γ-divergences at a small γ, taken as 10^{-3} in the text.
This is a definitions-and-identities paper. There is no new classification or generative benchmark. The facts that lock down are:
| Property | Ordinary JSD | Usual G-JSD | Extended G-JSD |
| Closed form on Gaussians | no | yes | yes |
| f-divergence | yes | no (in general) | yes |
| Square root a metric | yes | no | no |
| Bounded | ≤ log 2 | no | no |
The metric failure is explicit. On a three-point discrete triple, √G-JSD violates the triangle inequality by about 0.190 and the extended version by about 0.211. When the mean is arithmetic the two definitions coincide, because the partition is identically 1.
Anyone computing G-JSD between Gaussians can now write both the usual and the extended formula as Jeffreys plus Bhattacharyya and copy the expression. The real design choice is whether to normalize. Keep the extended version if coarse-graining stability of an f-divergence matters; keep the normalized version to match the existing G-JSD literature, and remember that they differ by Z−log Z−1.
Nothing here shows that the extended version is better inside a VAE or a GAN. G-JSD is already in use; this paper fills in the algebra of the geometric symmetrization.
There are almost no new experiments. Monte Carlo and the γ approximation stay at the method-discussion level. The triangle-inequality counterexamples are three-point discrete distributions; the size of the defect on continuous or high-dimensional Gaussians is not reported. The extended version is friendlier to positive measures, but densities in machine learning are usually already normalized, so the practical gain depends on meeting unnormalized models such as energy-based models. γ=10^{-3} is a suggested magnitude, not a measured error table. Because the square root is not a metric, it cannot be used as a kernel or embedding distance. Anyone hoping that swapping JSD for G-JSD would keep metricity is stopped here.