ResLRB: differentiable light tracing with constant 142 MB memory; naive AD OOMs past depth 16

Stochastic Graph Compression for Constant-Memory Differentiable Light Tracing

Linas Beresna, Eugene Fiume

SIGGRAPH Asia 2026 Technical Com

cs.GR

2026-10-08

ResLRB keeps one reservoir-sampled sensor connection per light path and replays it for backprop: differentiable light tracing peaks at 142 MB, where naive AD OOMs past depth 16.

What problem this solves

Differentiable rendering computes derivatives of a rendered image with respect to scene parameters such as materials, emitters, and geometry; it is the workhorse of inverse rendering. With a parameter space that large, derivatives have to come from reverse-mode automatic differentiation (AD): record every operation of the forward simulation into a computation graph, then walk it backwards once to deliver gradients to all parameters. In Monte Carlo light transport that graph holds every scattering event on every path, so it grows with path depth and sample count until the GPU runs out.

Viewpoint path tracing already has a fix. Path Replay Backpropagation (PRB) stores only the seed of a path's random number generator; the adjoint pass replays the same random numbers, regenerates the same vertices, and backpropagates on the fly, so memory stops scaling with depth. Light tracing breaks this. A path started at an emitter can connect to the sensor at every scattering vertex and splat flux onto the image, so a path of depth k yields up to k separate contributions, each with its own graph, and PRB's non-branching structure does not transfer. This four-page SIGGRAPH Asia 2026 Technical Communications paper from Simon Fraser University closes that gap.

Method

ResLRB (Reservoir Light Replay Backpropagation) keeps PRB's two-pass structure and inserts a stochastic compression step.

The primal pass does three things:

At each scattering vertex it connects to the sensor and streams the candidate (flux Ci, splat position ui, weight wi) into a single-item weighted reservoir (Vitter 1985): candidate i replaces the resident sample with probability wi/W.
The weight is luminance. Any strictly positive weight preserves unbiasedness, so luminance is purely a variance choice: the selection probability becomes proportional to each connection's share of the path's image contribution, and the value is already computed during the forward pass, so it costs nothing extra.
When the path ends, only the winning index I, the weight sum Wk, and the PRNG seed survive. Everything else is discarded, and no AD graph is ever recorded.

The adjoint pass re-seeds the PRNG and replays the path deterministically up to depth I. At each vertex it opens a throwaway AD scope for the local scattering computation; a log-derivative identity (the gradient of log f equals the gradient of f divided by f) turns the cached primal value CI into a multiplicative weight, so each vertex hands in its share of the gradient of CI and its small graph is dropped before the next vertex. Peak memory is bounded by the cost of differentiating one scattering event, independent of depth and connection count.

The estimator is (Wk/wI) times the winning splat. Taking the expectation over the reservoir selection recovers the sum of all k splats exactly, so the gradient is unbiased. The price is variance: only one connection per path is ever evaluated.

An attached extension handles the case where perturbing the parameters also moves the splat position, which is strongest at specular and refractive vertices (the glass-slab caustic is the canonical case). A forward-mode pass chains per-vertex Jacobians into a path Jacobian, cached at constant extra storage when the reservoir selects a vertex; rank-deficient Jacobians at perfectly diffuse vertices are regularized.

Results

Implemented as a custom integrator in Mitsuba 3 on top of Dr.Jit, evaluated on an NVIDIA RTX 5080 (16 GB) with the CUDA/OptiX backend, against naive reverse-mode AD (Mitsuba's ptracer):

Setting	naive AD (ptracer)	ResLRB
Peak memory, max path depth swept	Grows linearly, OOM past depth 16	Constant at 142 MB
Glass-slab caustic gradient, 32 spp	Runs	Matches the baseline up to Monte Carlo noise
Sample budget on the caustic	OOM beyond 32 spp	Reaches 1024 spp, same gradient, less noise

In the Grey & White Room scene (path length 32, 32 spp), gradients of an L2 image loss with respect to the floor's diffuse reflectance agree in sign and spatial structure. Time is what gets paid: replay traverses the path a second time, and Figure 6 charts computation time against depth.

Why it matters

Caustics and similar effects are exactly where light tracing with splatting earns its keep, and for image-space splatting there was previously no constant-memory way to backpropagate through it. ResLRB changes that with an unbiased gradient, so it drops into gradient-based optimization pipelines without baking bias into the objective. The mechanism travels, too: streaming reservoirs made their name as variance reduction in ReSTIR, and here the same tool is aimed at bounding a computation graph; any differentiable simulation whose graph explodes with branching is a candidate. Framed honestly, this is a four-page Technical Communication patching a specific gap in PRB, not a new paradigm.

Limitations

Variance goes up: k connections' worth of gradient per path is compressed into one sample. The paper guarantees correctness only in expectation, with no variance quantification and no analysis of how many samples per pixel it takes to match the all-connections gradient; the teaser figure uses 1024 spp to show a clean result, an implicit measure of the variance price.
Time for memory: replay adds a full extra traversal, presented as timing curves in Figure 6, and the overhead grows with depth.
Small evaluation: two scenes, one 16 GB GPU, a single baseline. No comparison against Radiative Backpropagation's time-for-memory trade-off and no end-to-end inverse optimization results.
The attached case leans on regularizing rank-deficient Jacobians at diffuse vertices, with no analysis of how regularization strength affects gradient accuracy.
The method assumes a path can be replayed deterministically from a PRNG seed; whether the structure survives cross-path reuse in the style of ReSTIR, or non-deterministic backends, is not discussed.

Terms

Differentiable rendering: derivatives of a rendered image with respect to scene parameters, powering inverse rendering.
Light tracing: Monte Carlo transport simulation that starts at emitters, as opposed to camera-side path tracing; strong on caustics.
Splatting: accumulating a path's flux directly onto sensor pixels from the emitter side.
PRNG replay: storing only a random seed and replaying the random sequence to regenerate a path, so no computation graph needs to be kept; the core of PRB.
Weighted reservoir sampling: keeping one sample from a stream with probability proportional to its weight; the basis of ReSTIR.
Caustic: bright pattern formed by light focused through refraction or specular reflection, the classic target scene of light tracing.

Source

What people are saying

ResLRB: constant-memory differentiable light tracing via stochastic graph compression ssh4net

Related papers

All paper explainers