Adapting Without Gradients: Affine Statistics Transport and What Its Certificate Can Tell You
Salim Khazem, Ibrahim Mohamed Serouis
cs.LG, cs.AI, cs.CV
2026-09-01
CASTER transports class Gaussians on frozen features, beating k-NN in 27 of 28 settings; gating turns -3.35 points into +1.69, but keeps only 0.6% of Tent's gain when reused.
Most test-time adaptation (TTA) updates network parameters at inference. Tent and EATA touch BatchNorm affine weights; SAR can also touch LayerNorm. That recipe fails when the accelerator has no backward pass, when a frozen binary or vendor API exposes no trainable weights, or when optimizer state blows the memory budget. The evaluation here makes the architectural catch concrete: the standard Tent/EATA implementations collect only BatchNorm parameters, so on ConvNeXt, ViT-B/16, DeiT-B, and Swin-T their eligible set is empty and the prediction is identical to the frozen source model.
CASTER studies a stricter regime. The feature extractor and the learned head stay frozen. Adaptation is an analytic move in feature space.
From labeled source data, CASTER builds a discriminative subspace from the generalized eigenvectors of between-class and within-class scatter. Dimensionality k is min(kmax, C-1, B-1): C-1 is the rank cap on between-class scatter, B-1 drops directions a target batch cannot support. Each class stores a projected mean, covariance, and prior. There is no source feature bank.
For a target batch it estimates one class-shared affine map: whiten with the source covariance, color with the target covariance, then shift means. When B < 2k it switches to a diagonal map so the degrees of freedom drop from k² to k. Source class Gaussians are pushed through that map and scored with a Gaussian discriminant. No network weight changes.
The map is not always safe. ImageNet-C has 1,000 classes and batches of 64, so most classes are absent and unconstrained transport wrecks the geometry. The transportability certificate inverse-maps confident pseudo-class centroids back to source coordinates and compares the residual to the minimum source-class margin, both in the same Mahalanobis metric, so a linear rescaling of features cannot change the ratio. If rt exceeds τ, or coverage is too thin, CASTER returns the frozen head. τ=0.8 is global, not tuned per dataset.
On frozen ImageNet-pretrained features, four backbones and seven datasets, CASTER beats k-NN on the same features in 27 of 28 cells. The one loss is ViT-B/16 on CIFAR-10, under 0.04 points. The largest gains are on Flowers-102, about 18–22 points on three backbones. Median retained state is 18× smaller; the k-NN bank on Tiny-ImageNet is 781 MiB. The inversion is reported: Flowers-102 stores only 1,020 training images, so CASTER's covariance state can be larger.
| Setting | Frozen head | Always transport | Certificate gate |
| 307 evaluation cells | 0 reference | −3.35 | +1.69 (τ=0.8) |
| ResNet-50, nine datasets | 59.05 | 55.70 | 60.05 |
| ImageNet-C | 0 | −21.2 | 0 (every batch rejected) |
Every cell that loses more than 10 points has certificate above 3.9. The bands overlap: a non-harmful cell can reach 8.9. The score detects catastrophe; it does not rank benefit. Inside the non-degenerate regime it correlates positively with gain, while the gate accepts low scores. On CIFAR-10-C, where catastrophic transport is absent, thresholding loses to transporting every batch. Any τ in [1.0, 3.9] yields +1.85 to +1.92, within 0.3 of the best point.
On corruption benchmarks the BatchNorm ResNets put Tent, EATA, SAR, and CASTER in a pile: CIFAR-10-C around 85.7–85.8 versus a frozen 72.0. On the four LayerNorm backbones Tent/EATA collapse to the frozen head (CIFAR-10-C 86.7); CASTER reaches 87.4 and SAR 87.3. On CIFAR-100-C LayerNorm, CASTER is 68.1 against frozen 65.1 and SAR 66.1. On ImageNet-C, Source-GDA loses 19.0, T3A loses 33.5, and SAR is the only adaptive method that still gains (+4.6).
The same certificate in front of Tent, on 57 matched CIFAR-10-C cells, admits 4.3% of updates and keeps 0.6% of Tent's available gain. A looser source-statistic referee admits 46.3% and still keeps only 32.3%. Acceptance rate anti-correlates with Tent's ungated gain (ρ=-0.86) and correlates with CASTER's own transport (ρ=+0.72). The failure is structural: the certificate reads the pre-update frozen representation, and Tent then changes that representation.
Cost on CIFAR-10-C fog, severity 3, ResNet-50: 1,076 img/s and 1,249 MiB peak, 1.6× Tent's throughput and 22.7% of its memory, 90% of frozen-head throughput.
This is a lightweight option when the model cannot move, and when the usual Tent configuration is inert on LayerNorm backbones. It is not a replacement for gradient TTA whenever a backward pass is available. On BatchNorm ResNets, Tent and EATA remain strong.
The reusable lesson is the certificate's scope. It is a safety floor for this transport, not a generic "should I adapt" score. Placed in front of a different mechanism, it blocks the batches that mechanism most needs.
CASTER inherits the frozen representation. Transport cannot recover class structure that is not already in the features. The affine, class-shared shift can fail under class-dependent drift or when batch composition is confounded with shift. The certificate is an empirical diagnostic, not a guarantee on arbitrary targets.
The Tent comparison is tied to BatchNorm-only parameter collection. It does not prove entropy minimization is impossible on LayerNorm networks. On Flowers-102 the storage advantage versus k-NN reverses, and the paper reports that inversion.