A Lightweight Differentiable Method for Real-Time Skin Rendering and Digitization
Mario Marquez-Balduque, Carlos Rodriguez-Pardo, Jorge Lopez-Moreno
cs.CV, cs.GR
2026-11-11
Differentiable SSSS fits Burley scattering from one light and view. On 225 patches from 45 people, masked FLIP is 0.08-0.09 and each patch finishes in 198s for real-time shaders.
Real-time skin shaders do not path-trace a volume. Subsurface scattering is a screen-space blur, and the knob on the shader is the Burley scattering distance: how far light travels under the surface. Offline inverse rendering recovers absorption and extinction, which do not plug into that knob. Hyperspectral, polarimetric, and light-stage captures can separate reflectance, but they stay in the lab, and published sets cluster on mid tones. Ears and cheeks also differ in thickness, so one global distance is the wrong model.
This paper estimates the shader parameters themselves. A consumer camera and a projector, in a dark room, with one light and one view, fit a scattering distance per facial region and blend those values into a profile map a runtime engine can read.
The projector sits about 20 cm from the skin at a 30 degree graze, which stretches the scattering halo. A Sony a6100 sits behind it at a calibrated 47 cm. Each patch is four frames: black, vertical stripes, inverted stripes, and a flat field. The black frame is subtracted. Stripes minus inverted stripes, thresholded with Otsu, become a hard shadow mask reserved for the last stage. Blurring that mask would rewrite the halo width. A lit-region mask is blurred with an 11 by 11 Gaussian so the scattering integral only sees illuminated skin, with projector defocus softened.
Geometry is not a structured-light mesh. A monocular depth map is scaled to 2 cm of relief. Normals come from a Sobel filter and stay frozen. Freeing them on a single view lets them steal gradient from the material.
The forward model is the Burley 2015 double-exponential profile. A texture value in [0, 1] maps to [0, 1] cm and is multiplied by 0.01 to reach meters, the same scale Unreal Engine 5 uses. Sample distances are 3D Euclidean distances from the depth buffer, so the profile does not shrink when focal length changes. One hundred twenty-eight Fibonacci samples draw radii from the inverse CDF of the profile. Transmission through thin skin uses inverted-normal ambient occlusion as a thickness guess and is drawn only in the final image. It is not optimized.
Adam runs the same three-stage schedule on every subject. Stage 1, 700 steps at 5e-3, fits a residual specular map and roughness. An L2 prior pulls roughness toward 0.6, the default skin roughness in real-time engines. Stage 2, 500 steps, unlocks albedo. The residual specular map keeps absorbing highlights and is discarded after convergence, so mirror energy does not land in albedo and then drag scattering distance. Albedo starts from k=2 clustering on LAB chroma after the brightest 10% and darkest 5% of pixels are removed, then a 2-sigma clipped mean. Index of refraction stays near 1.44. Scattering distance starts at 0.5. Stage 3, 600 steps at 1e-3, updates only scattering distance, and the loss moves onto the shadow mask. Direct light on the bright stripes would otherwise swamp the halo, especially on darker skin.
Each region gets one RGB scattering distance. Anatomical labels and the thickness map blend those scalars into a continuous texture. Differentiable screen-space subsurface scattering (SSSS) turns the per-pixel shader loop into a tensor integral: the image is replicated along the sample axis and fetched with bilinear filtering so the kernel matches the WebGL shader. Frames are center-cropped from 6020 by 4024 to 480 by 360.
The real set is 45 people, five facial regions, 225 patches, spread across the Monk Skin Tone Scale. Seventeen people (85 patches) were recaptured with three projectors. Errors below are inside the mask.
| Tone group | FLIP down | LPIPS down | SSIM up | PSNR up |
| Monk 1-3 | 0.093±0.026 | 0.086±0.020 | 0.615±0.066 | 28.61±1.68 |
| Monk 4-6 | 0.093±0.031 | 0.088±0.018 | 0.614±0.066 | 28.67±1.89 |
| Monk 7-10 | 0.077±0.021 | 0.115±0.015 | 0.503±0.072 | 29.65±1.35 |
Light and mid tones agree. On Monk 7-10, FLIP drops to 0.077 and PSNR rises to 29.65, while LPIPS rises to 0.115 and SSIM falls to 0.503. The paper attributes weak SSIM to luminance swings in shadow. It does not explain the LPIPS regression.
Figure 14 prints shorter distances on darker skin. On the paper's centimeter scale the annotated RGB curves are about 0.522/0.427/0.357 cm (light), 0.522/0.345/0.259 cm (mid), and 0.408/0.278/0.161 cm (dark). The direction matches stronger melanin. There is no spectrometer measurement behind it.
On an RTX 4070 Ti the three stages take 51.1 s, 38.3 s, and 109.0 s, 198.4 s in total, with a 9.4 GB peak.
The shadow mask is the clean ablation. Against a full-image loss, masked FLIP in the shadow falls from 0.11 to 0.06 (about 45%), and full-patch FLIP falls from 0.15 to 0.10. The full-image run bends the magnitude of scattering distance to soak up direct-light error. On synthetic patches with known Burley parameters, the fit returns to that ground truth, with example FLIP between 0.01 and 0.02. That is shader self-consistency. Three projectors versus one change the scattering distance by a mean L1 under 2%. Roughness gets sharper. Scattering barely moves. There is no head-to-head number against path-traced inverse rendering or hyperspectral fitting.
A production pipeline wants a distance map the existing skin shader already understands, not an extinction coefficient that someone must translate. The rig is a DSLR, a projector, and a dark room. Forty-five subjects cover the Monk scale, and the darker bins stay in the table.
The gain is incremental. Screen-space scattering is already the engine default. What changes is that the inverse problem is stated in those parameters, and the three-stage schedule separates albedo, specular, and scattering. One 480 by 360 patch costs about 198 seconds and 9.4 GB. That is an offline asset tool. It is not a face tracker.
Scattering is homogeneous inside a region. Monocular depth has no pores, so roughness comes out smooth. Nose cartilage and the ear helix never get their own single-scattering fit, and the missing transmission is explained by a longer scattering distance. Region blending keeps the look in a usable range. The size of that bias is not measured. Self-shadows in steep folds leak into albedo because visibility is not differentiable.
Error is measured under the same light and the same view. Synthetic ground truth is rendered by the same Burley shader. Angular checks sit in the supplement, with no numbers in the main text. The multi-light subset is 17 people, and three projectors still cover a narrow cone. Capture needs a dark room, locked exposure (ISO 100, 1/20 s, f/20), and a face pressed to a marker. The recovered distance is an effective shader parameter, not an intrinsic scattering coefficient. The paper says so.