RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models
Zijun Liao, Yian Zhao, Xin Shan, Yu Yan, Chang Liu, Lei Lu, Xiangyang Ji, Jie Chen
cs.CV
2025-10-29
A frozen DINOv3-ViT-B is aligned only to RT-DETR's deepest AIFI feature, with gradient-share gating. RT-DETRv4 hits 49.7-57.0 COCO AP at 273-78 FPS with unchanged inference.
Real-time detection keeps trading a thin backbone for feature quality. YOLO pushed the single-stage pipeline hard; RT-DETR was the first DETR that could fight YOLO on latency. Architecture tweaks after that pay less. Lightweight nets drop high-level semantics to hit frame rate. In RT-DETR's hybrid encoder, only the top map S5 goes through self-attention to produce F5. Cross-scale fusion and query selection both depend on F5, while the detection loss has to travel back through the decoder and CCFF. The paper calls this the F5 semantic bottleneck.
Vision foundation models such as DINOv3, CLIP, and MAE carry much richer semantics, and using one as the backbone blows the latency budget. The idea here is to pour that semantics into a lightweight detector at training time, then throw away the teacher and the projection head at inference.
The student stays an RT-DETR hybrid encoder: a CNN backbone emits S3/S4/S5, AIFI runs self-attention only on S5 to get F5, then a CNN fuses F5 with shallower maps. The teacher is a frozen DINOv3-ViT-B.
The Deep Semantic Injector (DSI) aligns only F5. Patch tokens are reshaped to a 2D grid and interpolated to F5's spatial size. A linear layer maps student channels into the teacher's space. The loss is negative cosine similarity at each location, matching direction rather than magnitude. Ablations are blunt: supervising backbone S3/S4/S5, or backbone plus F5, adds no AP; F5 alone adds 0.5. A linear projector beats 1×1 conv and MLP. Cosine beats MSE: 53.5 vs 52.7 on DEIM-M at 90 epochs.
A fixed distillation weight λ is the wrong knob. Gradient-guided Adaptive Modulation (GAM) tracks L1 gradient norms of backbone, AIFI, CCFF, and decoder each epoch, and looks at AIFI's share r. If r leaves [ρ−δ, ρ+δ], λ is rescaled toward the interval edge so AIFI's effective gradient share is pulled back. The best static λ (20) reaches 55.1 AP; GAM reaches 55.4 and converges faster early on.
DSI and the teacher are training-only. Parameter count and latency match the DEIM sibling of the same size.
COCO val2017, TensorRT FP16 on a T4.
| Model | Params / latency | AP | Main baselines |
| RT-DETRv4-S | 10M / 3.66ms | 49.7 | DEIM-S 49.0; YOLOv12-S 48.0 |
| RT-DETRv4-M | 19M / 5.91ms | 53.5 | DEIM-M 52.7; YOLOv12-M 52.5 |
| RT-DETRv4-L | 31M / 8.07ms | 55.4 | DEIM-L 54.7; YOLOv13-L 53.4 |
| RT-DETRv4-X | 62M / 12.90ms | 57.0 | DEIM-X 56.5; YOLOv12-X 55.2 |
That is 273 / 169 / 124 / 78 FPS. Versus DEIM the gains are 0.7 / 0.8 / 0.7 / 0.5 AP with identical structure and latency. In 36-epoch ablations DSI alone is +0.1 to +0.2 AP; GAM is what makes +0.5. DEIMv2, which uses DINOv3 as the backbone, is weaker at matched speed: v2-S 50.9 AP at 173 FPS against this M variant at 53.5 AP and 169 FPS, and it cannot cheaply attach a larger teacher.
This is a training-time free lunch, not a new detection head. If a production stack already ships RT-DETR, D-FINE, or DEIM, a retrain can hang a frozen VFM on the side and export the same engine graph. Teacher size and family are in principle swappable, so deployment is not locked to Tiny or Small DINOv3.
The gain is a steady 0.5–0.8 AP, not a generation of architecture. Whether it is worth the teacher forward pass depends on already living in this real-time DETR family, and on whether half a COCO point moves the product.
There is no Limitations section. The main tables use only DINOv3-ViT-B; the discussion names MAE and CLIP, with no swap table. Generality to CNN detectors is claimed; every number is on the real-time DETR family, with no YOLO run. ρ and δ for GAM are described, not fully swept. 36-epoch ablations and 50–120 epoch full runs should not be compared as one grid. Code is "coming soon", and extra train cost is a frozen ViT-B forward every step. The DEIMv2 comparison is restricted to similar latency; larger DEIMv2-X still scores higher, and is slower. There is no report on occlusion or extreme small objects, where a teacher prior could leak into the student.