Are the Latent Representations of Foundation Models for Pathology Invariant to Rotation?
Matouš Elphick, Samra Turajlic, Guang Yang
eess.IV, cs.CV
2024-12-17
On 363 TCGA-KIRC slides, 12 pathology FMs were rotated patch-wise. Rotation-aug models aligned better: PathDino m-kNN 0.85 vs Virchow 0.53; t-test p<0.0001.
A pathology slide has no upright. Tissue can land at any angle on the scanner, and a gland rotated 90 degrees is still that gland. Foundation models for digital pathology mostly ignore this. They tile H&E whole-slide images into 256×256 patches, encode each patch as a vector, and pass the bag to a slide-level head. If that vector moves when the patch rotates, the head is scoring a different sample.
Papers in this area report accuracy and cross-site generalization. Almost none measure what a 45-degree turn does to the latent space. Vision transformers have no rotational inductive bias: position codes sit on a grid, and self-attention does not know a pixel has been spun. Any invariance that shows up was learned, almost certainly from rotation augmentation. This paper measures how much of that learning actually happened.
The probe set is TCGA-KIRC: 363 H&E slides of kidney clear-cell carcinoma, read at 20×. Foreground is segmented in HSV, the five largest contours are kept, and 256×256 patches are taken from their bounding boxes when at least 75% of pixels are tissue. How many patches that produces is never stated.
Twelve frozen self-supervised pathology encoders are scored: Conch, Hibou (base and large), Kaiko (base and large), PathDino, Phikon, Phikon 2, Prov-GigaPath, UNI, Virchow, and Virchow 2. Each patch is rotated from 0° to 360° in 15° steps. The last-layer embedding is stored at every step. The unrotated embedding is the control.
Two alignment scores compare the control set with each rotated set. Mutual k-NN with k=10 asks what fraction of a patch's ten nearest neighbors survive the turn. Cosine distance asks how far the same patch's vector swings in angle. Tissue of the same type should stay close and point the same way.
Models are then split by one training-time choice: whether rotation augmentation was used. A t-test compares the two groups. Figure 2 marks the augmented models as triangles (PathDino, Kaiko-B/L, Hibou-B/L) and the rest as circles.
PathDino is the most stable on neighborhood overlap, mean m-kNN 0.85. Virchow is the least, 0.53. On cosine distance Hibou-L is tightest at 0.016; Phikon 2 is loosest at 0.145. The rotation-aug group beats the no-aug group on both scores: t = 6.91 for m-kNN and t = −8.88 for cosine, both p < 0.0001.
| Model | Rotation aug | Mean m-kNN ↑ | Mean cosine distance ↓ |
| PathDino | yes | 0.85 | 0.025 |
| Hibou-L | yes | mid (0.74) | 0.016 |
| Phikon | no | 0.80 | 0.07 |
| Phikon 2 | no | lower | 0.145 |
| Virchow | no | 0.53 | 0.11 |
The heatmaps in Figure 3 are worst at 45°, 135°, 225°, and 315°. The authors blame the corners: rotating a square changes which pixels sit at the border, so diagonal angles hurt more than 90°/180°/270°. PathDino reaches about 0.93 m-kNN on cardinal angles and still 0.83 on diagonals. Virchow drops to about 0.46 on some diagonal angles.
Size does not track invariance. Figure 2 scales markers by parameter count, up to about 1B. Virchow is one of the largest points and the worst m-kNN. Phikon 2 is newer and larger than Phikon and worse on both metrics. Virchow 2 improves a lot over Virchow, neighborhood overlap from 0.53 into the mid-0.7s and cosine distance from about 0.11 to about 0.03, even though Figure 2 still marks it as trained without rotation aug. That gain came from some other recipe change.
The two metrics also split. Hibou-L wins cosine distance and sits only mid-pack on m-kNN. Conch is nearly as tight as PathDino on cosine and near the bottom on neighbor overlap (0.66). Directional alignment and neighborhood geometry are not the same quantity.
Anyone freezing a pathology encoder for MIL should treat rotation augmentation as a filter. Slide orientation is not standardized in the clinic. If patch vectors wander with angle, mean pooling or attention pooling writes that wander into the slide statistic.
This is a measurement paper, not a new architecture. The finding matches how ViTs are built: rotation invariance is not free, and a cheap training-time augment closes much of the gap. PathDino was designed as a rotation-agnostic extractor (HistoRotate over 360°), so winning m-kNN is the expected outcome. Do not assume a newer or larger checkpoint is more stable. Phikon 2 is the counterexample.
One organ, one dataset, 363 slides. No downstream head is trained, so a high m-kNN is not evidence that grading or detection gets better. Patch count N is unreported.
The 45° dip is mixed with resampling artifacts. A square rotated off-axis loses corner tissue or gains padded pixels; that failure is not the same as "the model cannot recognize a rotated gland." The authors flag the corner pixels and do not separate the two.
The t-test is observational. The two groups also differ in pretraining data, SSL recipe, and size, so the correlation does not prove that adding rotation aug is sufficient. There is no controlled add/remove of the augment on a single model. Every model is a transformer, so the inductive-bias claim has no CNN or group-equivariant control. Stain, scanner, and magnification are left for later work, as the authors note.