Curved representational Bregman divergences and their applications
Frank Nielsen
cs.IT, cs.LG
2025-04-08
Nielsen shows curved Bregman centroids are right projections of the unconstrained mean; cosine is curved squared Euclidean, and alpha-spheres intersect by cutting the epigraph.
A lot of distances used in machine learning are not Euclidean. Clustering, contrastive learning, and fitting probabilistic models routinely use Bregman divergences: a family of asymmetric dissimilarities induced by a strictly convex generator F. Squared Euclidean distance and KL divergence both sit in that family.
In practice the parameters live on a curved subset. Vectors are normalized onto the sphere, probabilities sit on a simplex, covariances obey extra structure. The original formula still evaluates, but it is no longer a Bregman divergence, because the domain is no longer convex. Statistics already has a name for the analogous restriction of an exponential family: a curved exponential family. This paper imports that language into divergence geometry.
A curved Bregman divergence is the original BF, defined on Θ, restricted to a lower-dimensional, non-affine subset U. When the restriction is affine, the paper calls it a sub-dimensional Bregman divergence: the graph of F on that affine slice is itself a Bregman generator, so the restriction can be rewritten as a genuine Bregman divergence in fewer coordinates.
The cleanest example is squared Euclidean distance on the unit circle. With generator F(θ)=½⟨θ,θ⟩ and points constrained to (cos u, sin u), the formula collapses to 1−cos(u₁−u₂), the cosine dissimilarity. On the circle F is the constant ½, so it is not a Bregman generator, and cosine is not a Bregman divergence. It is a curved squared Euclidean divergence. The same construction works on the unit sphere in any dimension.
The main theorem: the barycenter of a finite weighted set under a curved Bregman divergence is the right Bregman projection of the unconstrained barycenter (the ordinary weighted mean) back onto U. The cosine centroid on the sphere, average then renormalize, is that projection.
The same picture covers several familiar constructions:
The application is intersecting α-divergence spheres. A Bregman sphere lifts to a hyperplane cutting the epigraph of F; the intersection of two m-dimensional Bregman spheres is an (m−1)-dimensional sub-dimensional Bregman sphere. Once α-divergences are representational Bregman, the same cutting-plane geometry applies.
This is a theory paper. There are no new downstream accuracy numbers. What it leaves is a set of identifications that can be checked.
| Object | Identity in this paper |
| Cosine dissimilarity | Curved squared Euclidean on the unit sphere |
| Discrete KL on the simplex | Sub-dimensional Bregman of extended KL on the positive cone |
| Jeffreys-Bregman centroid | Curved centroid on the curve (θ, ∇F(θ)) |
| Extended α-divergence | Representational curved Bregman under the α-embedding |
| Intersection of n α-spheres | Vertical projection of n hyperplanes cutting the α-epigraph, then cut against the α-representation of the simplex |
Curved centroids need not be unique. If the unconstrained mean of points on the unit circle lands at the origin, every point on the circle is a projection. The paper flags this with Fisher's normal circle model.
People who cluster, train contrastive models, or work in information geometry often treat cosine, KL, and α-divergences as separate families. The paper gives one projection picture: average in the ambient space, then project onto the constraint. Sphere normalization, simplex reparameterization, and symmetrized centroids all become instances of one theorem.
The more immediately usable piece is intersecting α-spheres. Smallest enclosing balls and nearest-neighbor structures were already written for Bregman divergences; α-divergences can reuse the same epigraph-and-hyperplane algorithms. The paper points at the smallest enclosing α-ball (the Chebyshev point) and at universal coding.
This is a geometric reorganization, not a new training method. No new model, no new benchmark. The value is putting divergences already in use onto one projection language.
There are almost no numerical experiments, and no comparison of curved centroids against existing approximations on a real clustering or representation-learning task. A separate section discusses approximating Jeffreys-type symmetrized centroids, without a complexity table or an error curve.
Non-uniqueness of the curved centroid is acknowledged; how an algorithm should pick among projections, and whether that choice is unstable, is left open. The claim that α-sphere intersection is efficient rests on the epigraph geometry. The paper reports no implementation, runtime, or numerical stability.
Future work is one sentence: information geometry of submanifolds of Hessian manifolds. That is still a layer away from the cosine and InfoNCE objectives used in everyday deep learning.