A closed-form formula for the Kullback-Leibler divergence between Cauchy distributions
Frédéric Chyzak, Frank Nielsen
cs.IT, cs.SC
2019-05-27
A Maple-assisted proof gives Cauchy KL as log[((s1+s2)^2+(l1-l2)^2)/(4 s1 s2)]. It is always finite, and swapping the two densities leaves the value unchanged.
KL divergence counts the extra nats spent encoding samples from p with a codebook built for q. Gaussians, and exponential families once the covariance structure lines up, already have closed forms. The Cauchy density \(p{l,s}(x)=s/(\pi(s^2+(x-l)^2))\) is a location-scale family with no mean and no variance. Its differential entropy is known: \(\log(4\pi s)\). The same-location, different-scale case was already written down in Nielsen's 2019 Entropy paper. Two Cauchys that differ in both location and scale did not have a clean formula.
Cross-entropy integrals can also diverge on heavy tails. Whether a closed form exists is a practical question: variational inference, robust clustering, and any loop that evaluates KL at every step cannot afford a quadrature inside the inner loop.
KL is cross-entropy minus entropy. Entropy is already known, so the remaining integral is
\[h^\times(p{l1,s1}:p{l2,s2})=-\int p{l1,s1}(x)\log p{l2,s2}(x)\,dx\]
Plug in the Cauchy density and this becomes \(\log(\pi/s2)\) plus a six-parameter definite integral
\[A(a,b,c;d,e,f)=\int{-\infty}^{\infty}\frac{\log(dx^2+ex+f)}{ax^2+bx+c}\,dx\]
Both quadratics must stay positive on the real line: leading coefficient positive, constant term positive, negative discriminant. For Cauchy the coefficients are just the four location-scale parameters.
The integral is messy by hand. Chyzak's Mgfun package in Maple runs creative telescoping: differentiate under the integral sign, build a total derivative that vanishes at the boundary, and turn the parametric integral into a linear ODE in the parameters. Four steps.
\[A=\frac{2\pi\bigl(\log(2af-be+2cd+\sqrt{4ac-b^2}\sqrt{4df-e^2})-\log(2a)\bigr)}{\sqrt{4ac-b^2}}\]
The appendix pastes the Maple session almost verbatim. Computer algebra proposes a closed form; differentiation and boundary conditions then pin down the integration constant. Prudnikov's table has a close cousin with two fewer parameters and no written proof; that derivation identifies Meijer G-functions.
The KL between two Cauchy densities is
\[\mathrm{KL}(p{l1,s1}:p{l2,s2})=\log\frac{(s1+s2)^2+(l1-l2)^2}{4s1 s2}\]
Four parameters, no special functions. When \(s1=s2\) and \(l1=l2\), numerator equals denominator and KL drops to 0, as it should.
| Setting | Formula | Comparison |
| General location-scale | \(\log[((s1+s2)^2+(l1-l2)^2)/(4s1 s2)]\) | No prior closed form |
| Shared location, different scale | \(2\log((s1+s2)/(2\sqrt{s1 s2}))\) | Recovers Nielsen 2019 |
| Shared scale, different location | \(\log(1+(l1-l2)^2/(4s^2))\) | New location-family case |
| Differential entropy \(h(p{l,s})\) | \(\log(4\pi s)\) | Matches the existing handbook |
The value is always finite. Swap \((l1,s1)\) with \((l2,s2)\) and the fraction is unchanged, so \(\mathrm{KL}(p:q)=\mathrm{KL}(q:p)\). The Gaussian location-scale family does not have this symmetry. The paper uses the normal family as the counterexample: the same translate-and-scale construction, yet Gaussian KL stays asymmetric.
The cross-entropy itself is \(\log[\pi((s1+s2)^2+(l1-l2)^2)/s2]\), which is not symmetric, because only \(s2\) sits in the denominator. Symmetry appears only after subtracting the entropy term.
Location-scale families also satisfy a scaling identity: \(\mathrm{KL}(p{l1,s1}:p{l2,s2})=\mathrm{KL}(p{0,1}:p{(l2-l1)/s1,\,s2/s1})\). That identity holds for any f-divergence, not only KL. Substituting the standard Cauchy into the closed form checks both sides.
The formula is short enough to type into a loss. Variational inference with Cauchy noise, heavy-tailed mixtures with Cauchy components, or a KL regularizer in robust regression no longer need a quadrature.
Standard practice treats KL asymmetry as a given, which is why forward KL and reverse KL induce different optima. Cauchy is a clean counterexample: no moments, yet the KL collapses to a symmetric log.
This is a short note, not a new algorithm. The engineering content is that a quantity that used to need numerical integration is now four arithmetic operations and a log.
There are no experiments. The paper never compares the closed form against adaptive quadrature for speed or accuracy, and never probes floating-point behavior when \(s\) is near 0 or the two distributions sit far apart. The argument of the log is positive over the reals, so the expression is theoretically stable; whether cancellation appears when \(s1\) and \(s2\) differ by several orders of magnitude is untested.
The proof depends on Maple's Mgfun and on the assuming mechanism for analytic branches of log and square root. The appendix is complete, but reproducing it needs that computer-algebra stack. Following the third-order ODE by hand is not pleasant.
The result covers univariate Cauchy only. Multivariate Cauchy, Student-t with \(\nu>1\), and logistic, all of which are heavy-tailed location-scale families, are left open. Symmetry is a Cauchy-specific fact; it does not extend to location-scale families in general.
The arXiv version is v2 from May 2019. Main text plus appendix run 8 pages. The formula that practitioners will actually use is one line.