How Much Data Is Enough? Uniform Convergence Bounds for Generative & Vision-Language Models under Low-Dimensional Structure
Paul M. Thompson
cs.LG, cs.AI, stat.ML
2025-12-29
CLIP-style prompt classifiers get uniform bounds scaling as Nd/ε² with class count and intrinsic dimension, not embedding width. Low ECE can hide worst-case miscalibration.
Medical vision-language models are used as decision support: an image in, a vector of disease probabilities out. Strong average accuracy does not mean every subgroup, rare condition, or confidence level is reliable. Average loss can be low and Expected Calibration Error (ECE, the binned average gap between confidence and accuracy) can look fine while the tails fail.
Paul M. Thompson (USC Stevens Institute for Neuroimaging and Informatics) asks a finite-sample question: under what structure can generative models and VLMs produce classifiers that are uniformly accurate and uniformly calibrated at sample sizes people actually have. Uniform here means a bound on the worst deviation across prompts, inputs, and subgroups, not a mean over the dataset.
Parameter-count heuristics do not match the empirical picture. CLIP trains on hundreds of millions of image-text pairs. Biomedical sets often sit at \(10^{5}\) to \(2\times 10^{5}\) annotated images. That is two orders of magnitude smaller. Domain-specific VLMs still classify reasonably at those sizes. The missing piece is a reason, and a precise meaning of "enough" in the uniform sense.
The object of analysis is not the full VLM parameter vector. It is the family of classifiers induced by varying prompts or class-prototype embeddings inside a restricted semantic space, with the vision encoder held fixed.
In CLIP-style models, the image embedding \(z(x)\) and the disease prompt \(pj\) live on the unit sphere. Class logits are a temperature \(\alpha\) times the inner product \(\langle z(x), pj\rangle\); probabilities are softmax. On the sphere the inner product is globally Lipschitz in \(pj\) with constant \(\alpha\). Softmax has a bounded Jacobian, and cross-entropy, Brier score, and smoothed calibration indicators are Lipschitz maps of those probabilities. The whole evaluation class is therefore Lipschitz in the prompt embedding.
Lipschitz plus a finite cover gives Lemma 1. Cover the prompt space with balls of radius \(\rho=\varepsilon/L\), concentrate at the centers, and pipe the error through the Lipschitz tube to every nearby prompt. Sample size satisfies
\[n(\varepsilon,\delta)\ge\frac{c}{\varepsilon^2}\bigl(\log N(\Theta,\varepsilon/L)+\log(1/\delta)\bigr)\]
so complexity tracks metric entropy of the parameter space, not ambient width \(D\). Metric entropy is the log of how many \(\varepsilon\)-balls you need; it is a geometric count, not Shannon entropy.
The extra geometric assumption is that disease prompts lie in a \(d\)-dimensional semantic subspace, \(d\ll D\). For \(N\) class prototypes, \(\log N(\Theta,\rho)\le c1 Nd\log(1/\rho)\), which yields
\[n(\varepsilon,\delta)\ge c2\Bigl[\frac{Nd}{\varepsilon^2}\log(L/\varepsilon)+\frac{1}{\varepsilon^2}\log(1/\delta)\Bigr]\]
A spectral version puts prompts in the ellipsoid of the embedding covariance. Only eigen-directions with \(\sqrt{\lambdai}\ge\varepsilon/L\) enter the cover. Sample size becomes \((c/\varepsilon^2)\) times \(\sumi\log(L\sqrt{\lambdai}/\varepsilon)+\) plus the usual \(\log(1/\delta)\). Fast eigenvalue decay means few distinguishable directions. The sum scales linearly in the number of class prototypes \(N\).
A DKW-style scaling is given as a heuristic, not a tight distribution-free bound: if outputs are Lipschitz in an \(r\)-dimensional latent, uniform deviations run like \(\sqrt{r/n}\), so tolerance \(\varepsilon\) asks for \(n\sim L^2 r/\varepsilon^2\) samples.
Hard ECE bins break Lipschitz at the edges. The paper switches to smooth gates or piecewise analysis away from boundaries; the dimension law stays. For patch-token transformer VLMs, bounded token norms and standard softmax attention let the same argument run in a low-dimensional text-token subspace, with different constants.
This is a theory paper. It reports no accuracy numbers against public classification baselines. What it does give is three explicit scalings and a judgment about current biomedical dataset sizes.
| Object | Sample scaling | Relative to |
| 1-D empirical CDF (DKW) | with prob. \(1-\delta\), sup deviation \(\le\sqrt{\ln(2/\delta)/(2n)}\) | independent of the distribution's shape |
| CLIP prompt classifiers (exact \(d\)) | \(n\ge c(Nd/\varepsilon^2)\log(L/\varepsilon)\) | independent of embedding width \(D\) |
| Spectral ellipsoid | \(n\ge(c/\varepsilon^2)\sumi\log(L\sqrt{\lambdai}/\varepsilon)+\) | only eigen-directions visible at scale \(\varepsilon/L\) |
| Heuristic | \(n\sim L^2 r/\varepsilon^2\) | \(r\) = intrinsic dimension |
DKW controls the largest gap between an empirical CDF and the true CDF on the real line. ECE is a binned average of a similar gap: a small DKW deviation implies a small ECE, up to bin width; the converse is false. That is the formal version of "average calibration can look good while rare or high-confidence regions fail."
The practical section compares \(10^{5}\)–\(2\times 10^{5}\) medical image-text pairs with the hundreds of millions used for CLIP. Under the low-dimensional and Lipschitz assumptions, the paper argues that current medical dataset sizes can already support uniform guarantees for \(N\)-way diagnostic models, and offers the same scaling as theoretical cover for conditional generation tasks such as style transfer and cross-modal synthesis.
It never plugs in concrete \(d\), \(L\), and \(\varepsilon\) to produce a number \(n\), and it never measures a real CLIP or MedCLIP embedding spectrum to evaluate the right-hand side. The result is a scaling law, not a filled-in data-budget table.
For medical VLM training, the bound changes the question. Parameter count and CLIP's hundreds of millions stop being the bar. Effective dimension of the prompt space, Jacobian or Fisher spectral decay, and the \(Nd/\varepsilon^2\) law become the quantities to measure.
For evaluation, the warning is specific. ECE and KL are averages. They do not control rare diseases or the high-confidence tail. Uniform reliability needs a DKW-style supremum, or at least separate reporting by subgroup and by extreme confidence bins.
For generative-model training, the claim is narrower. The bound governs evaluation functionals of classifiers induced by moving prompts on a low-dimensional manifold. It does not govern generalization of a VLM trained from scratch. Parameter count does not enter the bound directly, but model size can still matter through the Lipschitz constant \(L\) and through representation geometry. Reading this as "100k images suffice to train a medical CLIP" overreads the theorem.
The contribution is classical covering-number theory applied to VLM prompt classifiers. It turns "why do moderate datasets work" into a checkable scaling. It is not a new training method.
The paper draws its own fence: it does not claim that general VLM parameterizations are Glivenko–Cantelli. It only analyzes the induced family obtained by varying prompts inside a restricted semantic subspace. Non-uniform sampling, non-i.i.d. tests, and covariate shift would weaken the guarantees. Stein thinning, flow matching, and SHASH-style distributional models are mentioned as possible repairs; no bounds are given.
Three harder gaps sit in the writeup.
The low-dimension assumption is justified by "spectral structure of text and joint image-text embeddings." The body never shows its own eigenvalue plot and never reports a typical \(d\) for medical prompts. The right-hand side of the bound is missing its main input.
The parameter space is first written as the product of \(N\) unit spheres \((S^{D-1})^N\), then covered at a rate that depends on \(d\). The intended meaning is that prompts live on a \(d\)-dimensional submanifold of the sphere. The notation mixes the ambient sphere with the effective subspace; anyone reproducing the covering step has to insert that restriction by hand.
The jump from "uniform convergence of prompt classifiers" to "100k medical pairs suffice to train a medical VLM" never substitutes \(L\), \(d\), \(N\), and \(\varepsilon\) to get an \(n\). \(10^{5}\) is an order-of-magnitude judgment, not a numerical corollary. Hard-binned calibration is carved out of the Lipschitz argument, and hard-binned ECE is exactly what clinical papers report.
The analogy between the Hungarian construction (coupling the empirical process to a Brownian bridge) and normalizing flows is intuition. The paper itself notes the Hungarian lemma is one-dimensional and has no direct extension to neural flows.