Gaussian-JEPA: Joint-Embedding Predictive Learning for 3D Gaussian Splats
Bin Ren, Qi Ma, Yue Li, Zongyan Han, Yidi Li, Yuqian Fu, Rao Muhammad Anwer, Theo Gevers, Fahad Shahbaz Khan, Salman Khan
cs.CV
2026-08-16
Gaussian-JEPA predicts hidden 3DGS features without reconstructing primitives. Vs matched Gaussian-MAE, resampling drift falls 23.7% and R@1 at 55% missing rises 19.80 to 39.82.
3D Gaussian Splatting stores an object as anisotropic primitives: position, covariance, opacity, and appearance live on the same record. Real assets often hold tens of thousands of Gaussians; a practical encoder takes a fixed budget. Figure 1 makes the mismatch concrete. One 34.3K-Gaussian asset, independently subsampled twice to 1K primitives, is still the same object and a completely different observation. That variation is not a semantic augmentation. It is the interface between a dense asset and a fixed-size encoder.
Self-supervised work on Gaussian objects mostly follows Gaussian-MAE: mask local groups and decode centroids plus attributes back in input space. The target is the particular primitives that happened to survive that draw, and the decoder has to emit heterogeneous geometry and appearance channels. Contrastive learning skips reconstruction, but the invariances track whichever views and augmentations you picked, and global agreement never supervises missing spatial content.
Gaussian-JEPA trains the encoder to predict representations of held-out token blocks from visible context. The JEPA idea is familiar from images and point clouds (I-JEPA, Point-JEPA, 3D-JEPA). A Gaussian token is harder: geometry, anisotropic support, visibility, and appearance are coupled, and missing evidence can span different spatial extents.
Pretraining uses the ShapeNet55-GS assets released with Gaussian-MAE: 51,934 training objects. Each asset is buffered to 8,192 entries, then 1,024 Gaussians are drawn without replacement. Every primitive is 14-D: centroid 3, opacity 1, scale 3, rotation quaternion 4, DC spherical-harmonic coefficients 3.
Tokenization follows Gaussian-MAE grouping. FPS on centroids picks 64 group centers; KNN gathers 32 neighbors each; a PointNet-style tokenizer emits 64 tokens of width 384. Neighborhoods depend only on centroids; token content uses all 14 channels.
Those 64 tokens are split into four non-overlapping target blocks with sizes [11, 9, 7, 5] (32 tokens) and a shared 32-token context. Blocks are sampled large-to-small by nearest neighbors, so the mask ratio stays at 50% while spatial support varies. Non-overlap is at token-center indices. Underlying primitives can still appear in more than one neighborhood.
Three pathways run per iteration:
Prediction is Smooth L1 in both spaces. The projectors are trained by a separate feature-space grounding loss (VISReg on each branch, plus a cross-covariance penalty). The total objective is Lpred + 0.1 Lground. EMA momentum rises linearly from 0.996 to 0.9999.
There is no input-space decoder and no reconstruction term on any Gaussian attribute. The encoder is a 12-layer Transformer, width 384, 6 heads, trained 300 epochs with batch 256 and AdamW on H200. Downstream keeps only the online encoder at 21.83M parameters, matched to Gaussian-MAE. Pretraining is 21.9% slower per step and uses 48.6% more peak memory, because the EMA encoder runs the full token field once per target block.
The matched baseline is Gaussian-MAE under identical Gaussian inputs and evaluation recipes. Point-cloud numbers are cited under their original protocols and are not treated as matched.
Resampling consistency uses all 2,467 ModelNet40-GS test objects, unseen in pretraining. Five independent 1K samples per object: one gallery, four queries. Relative drift divides same-object distance by that encoder's own mean distance to other objects.
| Method | Rel. drift ↓ | R@1 (%) | R@5 (%) |
| Gaussian-MAE | 0.3400 | 92.95 | 98.89 |
| Gaussian-JEPA | 0.2594 | 93.00 | 99.36 |
Relative drift drops 23.7%, and 96.2% of test objects move less. R@1 barely changes, so the stability is not a collapsed embedding.
Partial observation removes a spatially coherent cap of groups. At 55% missing groups, retrieval against the full 2,467-object gallery:
| Method | R@1 | R@5 | R@10 |
| Gaussian-MAE | 19.80 | 39.02 | 49.14 |
| Gaussian-JEPA | 39.82 | 65.28 | 75.95 |
R@1 is 20.02 points higher. The gap is already 10.56 points at 30% missing and 13.36 points at 70%. Paired bootstrap 95% intervals exclude zero. This is the paper's strongest result.
Shape completion on ShapeNet55-GS crops a half-space to 512 Gaussians, freezes the encoder, and trains the same decoder to emit a full 1K sample. Chamfer distance falls from 0.0732 to 0.0678 (7.4% relative), F1% rises from 6.62 to 7.42, and at 50% visibility PSNR goes 16.03 to 17.24 dB with SSIM 0.7148 to 0.7469. Three decoder seeds agree on the direction.
Semantic transfer is incremental. ShapeNet-Part class mIoU is 84.5 versus 84.2 for Gaussian-MAE. Full fine-tuning on ModelNet: 94.94 vs 94.16 on MN10 (+0.78) and 92.63 vs 92.54 on MN40 (+0.09). Frozen-backbone gaps are larger: +1.50 linear and +2.55 MLP-3 on MN40. Ablations show dual projections plus feature-space grounding lift resampling R@1 from 90.85 (single space) to 93.23; swapping grounding for attribute reconstruction drops R@1 to 88.18.
For people indexing, completing, or reading frozen features from object-level 3DGS assets, the practical takeaway is simple: do not pin the pretext to the raw attributes of one 1K draw. Fixed encoder budgets against dense assets are the default, not a lab trick.
Do not read this as a ModelNet leaderboard paper. Classification and part segmentation only edge out matched reconstruction. Point methods often post higher numbers under their own recipes, and the paper marks those with a dagger. Gaussian-MAE's published headline also used 1K pretraining with 4K transfer; the main tables lock both methods at 1K, where MN40 full fine-tuning is essentially tied.
"Decoder-free" means no input-space decoder and no attribute reconstruction during pretraining. A training step on H200 is still slower and hungrier. The saving shows up downstream, not on the pretraining invoice.
The authors stay at object-level assets with a fixed 1K budget. Large scenes, dynamic Gaussians, and adaptive budgets are left open. Resampling tests random subsets of one source asset; whether two independently optimized Gaussian reconstructions of the same mesh should match is a different question. Inputs remain 14-D with DC-only spherical harmonics, and grouping still keys off centroids. The geo and app branches have no geometry or appearance labels.
ModelNet full fine-tuning is noisy. In the supplement, Gaussian-MAE's published 1K-to-4K MN40 number is 93.35; their official-code rerun is 92.46; Gaussian-JEPA at 4K transfer is 93.21. The main 1K-transfer MN40 gap of 0.09 cannot support a classification claim. Completion F1% is still 7.42 in absolute terms; the relative gain does not mean a shippable completed asset. Pretraining is ShapeNet synthetic Gaussians only. Scanned real-world Gaussians are not in the primary tables.