RGBD20K: A Large-Scale Benchmark for RGB-D Semantic Segmentation
Shaohua Dong, Zexuan Meng, Haiyan Sun, Bing Fan, Cuicui Zhang, Dylan Joseph, Kewei Sha, Yunhe Feng, Heng Fan
cs.CV
2026-09-24
UNT assembles existing indoor RGB-D sources into RGBD20K (20,000 pairs, 160 classes). Score-purified fusion hits 46.3 mIoU there, 0.9 above same-backbone GeminiFusion.
Indoor RGB-D semantic segmentation, pixel labels from a color frame plus a depth map, has lived on two small benchmarks for years. NYUv2 has 1,449 pairs and 40 classes. SUN RGB-D grew the image count to 10,335 and kept 37 classes. That is thin fuel for a ViT, and it cannot name most objects in a real room.
The labels are noisy as well. SUN RGB-D drops instances, mixes categories, and paints sloppy boundaries. Matterport3D holds 194,400 RGB-D frames and ScanNet is large too, but both target 3D reconstruction or a handful of structural classes, not pixel-accurate 2D masks. A University of North Texas group stitched existing indoor RGB-D sources, relabeled them under a 160-class taxonomy, and released RGBD20K.
The images were not newly captured. The 20,000 pairs come from seven public sources: 10,335 from SUN RGB-D, 8,365 from DIML RGB-D (originally a pre-training set), plus subsets of RGB-D Mirror, VidSOD, DepthTrack, RGBD1K, and ARKitTrack, about 1,300 frames in total from the last five. Everything is mapped onto 160 classes and 75 indoor scene types, 368,312 instance masks, 18.4 per image on average. The split is 18,000 / 2,000, stratified so every class appears on both sides and the long tail is kept on purpose.
Annotators work pixel-wise in an interactive tool, from coarse groups such as furniture down to table types and device types. Parts attach to a parent when they exist, a drawer under a cabinet. Occlusions follow depth order: walls and floors sit at the back, nearer objects cover them. The paper says every pair went through several manual passes. It does not report inter-annotator agreement.
The model is score-purified fusion (SPF). The skeleton matches GeminiFusion: a four-stage hierarchical encoder, Swin-L-384 backbone, SegFormer head. The change is the fusion block. Cross-modal attention usually has to suppress sensor noise, fix misaligned depth edges, and gather long-range context at the same time, so the weights get diluted by unreliable locations. SPF's rule is purify first, attend second.
Three steps:
Training runs 300 epochs on four H100 GPUs. The learning rate steps down every 100 epochs from 6×10⁻⁵ to 1.5×10⁻⁵.
Annotation noise is measurable. Relabel the SUN RGB-D test set only, leave training untouched, and a pretrained DPLNet rises from 52.8 to 55.0 mIoU. The evaluation protocol itself was eating 2.2 points. Retrain and retest on the relabeled 37-class set (SUN RGB-D†) and DPLNet reaches 59.0, 6.2 above the original. Every method in the table moves up on †.
RGBD20K is much harder. The same models sit in the 53–60 band on NYUv2 and fall to 22–46 on the 160-class set.
| Method | Params | NYUv2 | SUN RGB-D | SUN RGB-D† | RGBD20K |
| PGDENet | 100.7M | 53.7 | 51.0 | 54.6 | 22.4 |
| DPLNet | 88.6M | 59.3 | 52.8 | 59.0 | 28.0 |
| DFormer v2 | 95.5M | 58.4 | 53.3 | 59.7 | 44.1 |
| GeminiFusion | 369.2M | 60.2 | 54.6 | 61.5 | 45.4 |
| SPF | 416.4M | 60.5 | 55.0 | 62.2 | 46.3 |
SPF leads on all four columns. Against GeminiFusion, which shares the same backbone family, the gaps are +0.3 / +0.4 / +0.7 / +0.9, at +47M parameters. Drop the score term and RGBD20K mIoU falls to 45.5 (−0.8). Purifying only Key or only Value, or only RGB or only depth, never recovers 46.3. Learnable noise beats random Gaussian noise by about 1.0. DPLNet scores 59.3 on NYUv2 and 28.0 on RGBD20K; copying its original prompt-tuning recipe onto 160 classes breaks down, and the paper does not say why.
For indoor robotics and embodied perception, this is one of the finest-grained public RGB-D segmentation sets. 160 classes split "table" into types and keep the long tail, closer to a real room. The data lives in the accompanying GitHub repo.
The model is an incremental patch. SPF adds a reliability gate on top of GeminiFusion and wins every column by a little. 0.3–0.9 mIoU against 47M extra weights is a weak engineering trade. The relabeling study is the more useful result: a slice of published benchmark scores is annotation noise, not model quality.
SOTA on 160 classes is still 46.3. Fine-grained indoor RGB-D segmentation is not solved. The paper mentions open-vocabulary and zero-shot use and does not run those experiments.
There is no limitations section. Several gaps sit in the open.
The set is a remix, not a new capture. Half the frames come from SUN RGB-D, about 40% from DIML, and the tracking plus saliency extras add up to under 1,300 images. Scene diversity is bounded by those original capture protocols. Everything is indoor. There is no outdoor split and no dedicated low-light subset. Matterport3D and ScanNet are larger; the paper correctly notes they serve different tasks with coarser labels. "Large-scale" here means large among fine-grained 2D RGB-D segmentation sets.
Annotation quality is argued with side-by-side figures and downstream mIoU, not inter-annotator agreement or a sampled error rate. Under a 160-class long tail, rare classes weigh heavily on mean IoU. Per-class numbers are missing, so it is unclear whether the tail was learned or averaged away.
SPF's lift over GeminiFusion is small at a higher parameter count, with no latency or FLOPs. DPLNet collapsing to 28.0 on RGBD20K leaves a question about whether every baseline was reproduced under a fair recipe. The test split is 10%. Stratification puts every class in the test set; tail classes may still have very few test pixels.