Gaussian Processes and Reproducing Kernel Hilbert Spaces: Connections and Equivalences
Motonobu Kanagawa, Philipp Hennig, Dino Sejdinovic, Bharath K. Sriperumbudur
stat.ML, cs.LG, math.NA, math.PR, math.ST
2025-06-20
This monograph shows GP and RKHS methods coincide for interpolation, regression, quadrature, and MMD/HSIC, via the isometry between the Gaussian Hilbert space and the RKHS.
Positive definite kernels split machine learning into two nearly parallel traditions. One treats the unknown function as a Gaussian process sample, puts a prior on it through a covariance kernel, and computes a posterior mean and posterior standard deviation. The other searches a reproducing kernel Hilbert space for a smooth function that fits the data: kernel ridge, kernel interpolation, MMD, HSIC. Both use the same kernel. Equivalences are known for particular tasks. What has been missing is a single account of why they hold, and whether the RKHS side can inherit GP-style uncertainty quantification.
This is a Cambridge-style monograph, expanded from a 2018 technical report. The intended readers are researchers and students on either side of the split. The aim is to make theorems transferable, not to invent a third vocabulary.
The unifying object is the canonical isometry between the Gaussian Hilbert space (GHS) of a zero-mean GP and the RKHS of its covariance kernel. The isometry sends the canonical feature k(·, x) to the random variable F(x) and preserves linear combinations and inner products. Estimating a quantity of interest by projecting it onto the data subspace is then the same operation on both sides: in the GHS for GP methods, in the function space for RKHS methods.
Chapter 4 defines linear functionals of a GP (point evaluation, integrals, derivatives, stochastic integrals) as the GHS element that corresponds to the functional's Riesz representative in the RKHS. A functional is well-defined for the GP if and only if it is continuous on the RKHS of the kernel. Theorem 4.9 gives the generic equivalence: the root-mean-square of a linear functional of the GP equals the maximum absolute value of that functional over unit-norm RKHS functions. Concrete equivalences for interpolation, regression, and quadrature grow out of this identity.
Regression is rewritten as interpolation with a regularized kernel: the original kernel plus the noise covariance. When the KRR penalty is noise variance over sample size, the estimator coincides with the GP posterior mean, and the optimal convergence rates line up. The distribution chapter rewrites MMD as the RMS difference of a GP sample's means under two distributions, and HSIC as the expected squared covariance of GP transformations, so kernel quadrature and Bayesian quadrature become two readings of the same algorithm.
There is no new leaderboard. The deliverable is a set of equivalences that belong in the working toolkit.
Interpolation: the GP posterior mean equals the RKHS interpolant. The posterior standard deviation equals the RKHS distance from the unknown point value to the data subspace, and equals the worst-case error over unit-norm RKHS functions. RKHS interpolation therefore carries uncertainty quantification for free.
Regression: the GPR posterior mean equals the KRR estimator. Posterior variance plus noise variance equals the squared worst-case error in the RKHS of the regularized kernel. The minimax rate for the GP posterior mean (van der Vaart and van Zanten) matches the KRR rate (Fischer and Steinwart) in the corresponding regime. The apparent contradiction (a GP sample lies outside its native infinite-dimensional RKHS with probability one) is resolved by placing the sample in a slightly larger RKHS almost surely, under conditions reviewed from Steinwart (2019).
| Task | GP side | RKHS side | Link |
| Interpolation | posterior mean / std | kernel interpolant / worst-case error | GHS–RKHS isometry |
| Regression | GPR posterior mean | KRR | interpolation with a regularized kernel |
| Integration / sampling | Bayesian quadrature | kernel quadrature, herding, thinning | GP reading of MMD |
| Dependence | expected squared covariance of GP maps | HSIC | two norms of one functional |
GP practitioners usually have uncertainty and lack a vocabulary for distances on distributions. Kernel practitioners usually have MMD and HSIC and treat posterior variance as someone else's object. The monograph pins both vocabularies to the same projection. In practice, a KRR predictive interval can be read as a GP posterior variance; a Bayesian quadrature design criterion can be read as an MMD worst-case error. The modeling assumptions do not fight: choosing a kernel chooses smoothness for both GP samples and RKHS functions. Matérn α and the infinitely differentiable squared-exponential kernel sit on one spectral line.
This is a unifying proof monograph, not an algorithms paper. There is no new benchmark, no implementation bake-off, and no complexity main theorem. The isometry and the equivalences target linear estimators and linear functionals; deep kernels, learned kernels, and non-Gaussian processes sit outside the main narrative. The fact that GP samples almost surely miss the native RKHS remains true; membership in a slightly larger RKHS needs extra regularity and is not a footnote one can drop. Rate alignment requires the KRR penalty to be noise variance over n; a freely tuned λ breaks the equivalence. For an engineer who only wants to tune GP hyperparameters, the book is dense at about 160 pages, and Chapter 2 is not optional.