DyRAD: Radar Novel View Synthesis for Dynamic Driving Scenes
Merav Keidar, Tomer Borreda, Rajalakshmi Nandakumar, Or Litany
cs.CV
2026-09-30
DyRAD reconstructs dynamic scenes as point reflectors and renders full range-azimuth-Doppler tensors with a fixed sensor PSF: 90.7% detection hit rate on RADIal vs 26.9% for baselines.
Closed-loop evaluation of autonomous driving stacks needs reconstructed scenes that can be re-rendered from viewpoints the vehicle never drove. Cameras and LiDAR got there through NeRF and Gaussian-splatting pipelines for dynamic driving scenes; radar did not. Existing radar novel-view synthesis either handles dynamic scenes but reconstructs only range-azimuth maps, discarding Doppler (radar's only direct velocity measurement), or renders Doppler under a static-scene assumption where all motion is ego motion. A radar measurement is also not a picture of the scene: the FFT and beamforming chain smears a single point reflector across many bins, and existing representations absorb that smear into scene geometry, which then renders incorrectly from displaced viewpoints. And prior evaluation only holds out frames along the driven trajectory, wedged between nearly identical training poses, which tests interpolation rather than the displaced views closed-loop simulation actually needs.
The paper exposes a measurement problem along the way: about 97% of RADIal bins sit at the noise floor, and a constant-prediction control scores higher SSIM than every reconstruction method and higher PSNR than both baselines. PSNR and SSIM say little about structure in sparse radar data, so the evaluation leans on Pearson correlation and detection metrics.
DyRAD, from Technion, NVIDIA, and Cornell Tech, pairs a point-reflector scene with a frozen sensor model.
The reason for zero-extent reflectors and a frozen PSF: allow learned reflector width or learnable PSF bandwidth, and the optimizer uses that blur to compensate misplaced reflectors, re-entangling scene structure with sensor effects. Freezing the PSF also buys sensor-configuration transfer: swap the PSF and sampling grid, keep the scene, and render under a different configuration without refitting.
| Setting | Metric | DyRAD | Best baseline |
| RADIal on-path | Foreground hit rate | 90.7% | 26.9% (RadarFields) |
| RADIal on-path | Full-RAD correlation ρ | 0.272 | 0.068 (RadarSplat) |
| RADIal on-path | Object-region ρ | 0.658 | -0.011 |
| RADIal, 2 m lateral round trip | Hit rate | 91.7% | 18.1% |
| Boreas on-path | Object-region RA ρ | 0.622 | 0.178 (RadarSplat) |
| Synthetic off-path | Object-region RAD ρ | 0.356 | 0.086 (RadarFields) |
Hit rate is the fraction of annotated boxes with reference detections that also contain predicted detections. Boreas uses a 360-degree spinning radar with no Doppler, the baselines' native setting; DyRAD still wins object regions there, so the gains do not come purely from Doppler supervision. Even the fully static ablation reaches 0.533 RA correlation against 0.291 for RadarSplat in the baselines' native RA output.
Ablations: Doppler supervision cuts vehicle Doppler peak-location error from 1.843 to 0.732 bins. The fixed PSF scores off-path detection F1 of 0.236 versus 0.074 for learned extents and 0.124 for learned bandwidths, with recall more than doubled. Configuration transfer: a model trained on measurements with 4x-coarser range and 2x-coarser Doppler resolution, rendered directly with the fine PSF, cuts detection Chamfer distance from 7.27 to 3.20 m and lifts F1 from 0.101 to 0.195 over linear upsampling of the coarse renders. Cost is modest: 7.2 minutes for 10k training steps on an RTX 4090, 5.65 ms per full RAD tensor, about 1 GB peak memory.
Two honest misses: on synthetic off-path data DyRAD averages 4.9 false-positive boxes per scene versus 0.7 for the static variant, so coverage is bought with false alarms; and in configuration transfer, linear upsampling scores higher on most correlation and SSIM metrics, with direct rendering winning on detection and object-region PSNR.
For simulation teams, radar finally combines dynamic scenes, displaced viewpoints, and Doppler rendering in one method; the radar channel of closed-loop simulation no longer has to fall back to static replay. Decoupling the sensor response from the scene generalizes beyond radar to any reconstruction where the measurement is the scene convolved with a sensor response, and no-refit configuration transfer is the immediate payoff. The off-path evaluation protocol (cycle consistency on real data plus a released synthetic benchmark with displaced ground truth) raises the bar for a subfield that has been scoring interpolation.
Temper expectations too: this is the first complete assembly of known parts (point primitives, rigid tracks, spherical harmonics, PSF convolution) applied to radar's dynamic case, and the individual pieces are not new. The value is in the combination and the validation.
The authors list four: reflector-to-object assignments come from annotations and stay fixed during optimization, so initialization errors cannot be corrected; multipath and speckle are unmodeled, limiting realism; the implementation is ground-plane only because the data lacks elevation; configuration transfer covers processing configurations of one physical radar, not different sensors.
Reading closely raises more: dynamic objects depend on bounding-box annotations, and unannotated classes such as pedestrians or debris are unaddressed; RF4D, the actual dynamic-scene baseline, was excluded because its code is "still under checking", so every reported baseline is a static method with an ego-motion Doppler lift, and the 90.7% vs 26.9% gap should be discounted accordingly; real-data off-path has no ground truth, so cycle consistency is a proxy and its scores are not comparable to on-path numbers; all experiments run on 50-to-60-frame sequences, with no large-scale validation.