LumiTokens Relights 3D Scenes in Token Space, 27.76 dB with 8 Views

LumiTokens: 3D Relighting via Token-Space Lighting Transformation

Yiwen Chen, Matheus Gadelha, Huaizu Jiang

cs.CV

2026-08-19

LumiTokens edits 3072 scene tokens with Plücker light rays. Novel-view PSNR is 27.76 from 8 views, above 50-view TensoIR at 26.17; extra lights compose in token space.

What problem this solves

Relighting a multi-view capture under new illumination currently splits into three families, and none of them is built for "add a key light, look, then add a fill."

Inverse rendering recovers normals, albedo, and roughness, then re-evaluates a rendering equation. Cross-view consistency comes for free from the 3D parameterization, but the pipeline wants dense captures, per-scene optimization, and a BRDF that cannot express subsurface scattering or rich inter-reflections. Diffusion relighting skips the material split and looks more natural. Each new light still needs a full generative pass, and there is no persistent scene state. Hybrid methods such as RelitLRM and NeAR feed-forward a 3DGS and still regenerate appearance through a generative branch when lighting changes.

LVSM and SRT already compress posed multi-view images into 1D tokens with no fixed physical meaning, then decode novel views. LumiTokens asks whether those tokens can be edited for lighting, rather than only decoded.

Method

The backbone matches LVSM: a 12-layer encoder, a 12-layer decoder, token width 768, 3072 scene tokens per scene. A Scene Token Editor with the same depth sits in between. Four to 32 posed images are patchified with Plücker camera rays and mixed, via self-attention, with a bank of learnable empty tokens that become the scene representation.

All lights become Plücker ray tokens. Each ray carries direction, moment, color, and radiant power.

• Environment maps: distant lighting. The moment is zeroed, leaving a pure direction.
• Point lights: rays originate at the source and point at the scene, so direction and moment jointly encode the 3D location.
• Area lights: origins are sampled on the emitting surface, encoding both position and extent.

The editor concatenates scene tokens and light tokens and runs self-attention. Cross-attention is deliberately avoided: light tokens need to look at the scene if shadows and highlights are going to land in the right place. Updated light tokens are dropped. Updated scene tokens go to the decoder.

The decoder queries those tokens with target-camera Plücker rays. LVSM's per-token MLP readout is replaced by a DPT head that reassembles tokens into multi-scale feature maps and convolves them into a full image. The same frozen decoder renders both original and edited tokens, which is the practical test that edits stay near the learned manifold.

Training freezes the pretrained decoder and finetunes encoder plus editor. The objective is L2 + LPIPS rendering loss, plus a lighting-invariance loss: encodings of the same object under two lights should match, so the encoder keeps geometry and materials and leaves illumination to the editor. To support incremental lights, training samples a random chain of sources, edits repeatedly in token space, and applies the rendering loss at every step against Blender ground truth.

Data: 20k filtered Objaverse objects, Blender Cycles at 512², 80 upper-hemisphere views, 16 lighting setups (4 HDR maps, 4 point, 2 area, 2 mixed), about 20 million images. Evaluation uses 900 held-out objects under 8 unseen lighting conditions.

Results

Multi-view relighting takes 4 input views and relights at the original poses, with least-squares exposure alignment following LightSwitch.

PSNR beats Neural Gaffer by 2.1 dB. LPIPS loses to Neural Gaffer's 0.030; a diffusion prior still wins on perceptual sharpness. The paper states SSIM and LPIPS stay identical under scene-level rescaling, with PSNR dropping only 0.12 dB. The SLR row in the table does not match that sentence, so the ILR numbers are the ones to cite.

Novel-view relighting uses 8 inputs and 8 held-out cameras.

Eight views and no per-scene optimization beat TensoIR's 50-view PSNR. SSIM is slightly worse (0.917 vs 0.928); inverse rendering still owns structural precision. Real data is mixed. On Stanford-ORB, LightSwitch reaches 32.02 PSNR against LumiTokens' 31.01. On Objects-with-Lighting, LumiTokens is 26.76 versus TensoIR's 26.12.

Freezing the encoder yields 25.06 PSNR on novel-view relighting. Unfreezing reaches 26.12. The invariance loss reaches 27.05. An auxiliary albedo head hits 27.09 PSNR but worse SSIM/LPIPS (0.934 / 0.057 vs 0.952 / 0.045) and needs albedo labels. End-to-end on an A100 is 3.79 s, against 16.85 s for Neural Gaffer, 532 s for LightSwitch, and 3953 s for TensoIR.

Chaining edits in token space holds quality for the early steps of a 10-step protocol. Decode-and-re-encode in pixel space drifts from step one. Multi-step supervision during training is required; a single-step editor walks off the manifold on long chains.

Why it matters

Latent scene tokens gain an operation they did not have: persistent, composable lighting edits. Product scans, catalog shots moved into a customer's room, and AR objects dropped into real lighting all need a light change without a rescan. The method does not promise a physically correct BRDF. It does promise sparse inputs, a feed-forward pass, and extra lights without another diffusion run.

The people who can pick this up today are teams already sitting on LVSM-style token scenes. The training set is synthetic PBR objects. This is not a general indoor relighting system yet.

Limitations

The authors flag the obvious next gaps: real captures, complex materials, imperfect poses. RelitLRM weights were unavailable and NeAR's code landed after the experiments, so there is no comparison to the strongest hybrids. On Stanford-ORB, LightSwitch with 16 views wins PSNR. The real-data lead is not established.

"Edited tokens remain in the same latent space" is inferred from render quality and multi-step training, not from a direct manifold metric. Self-attention can paint inter-object shadows and reflections, but without a rendering equation those effects may be memorized rather than computed, especially for extreme materials and multiple bounces. Cite the ILR table, not the SLR row, until the paper's own numbers agree with its prose.

Terms

• Plücker coordinates: a 3D line as a direction plus a moment vector; environment, point, and area lights share one ray format.
• Scene Token Editor: a transformer that self-attends over scene tokens and light-ray tokens and emits relit scene tokens.
• Progressive relighting: stacking another light in token space without decoding back to pixels between edits.
• DPT head: a dense readout that folds decoder tokens into multi-scale maps, sharper than a per-token MLP.
• Albedo-exposure ambiguity: albedo and exposure can trade off and still look plausible, so metrics use a least-squares scale first.

Source

• arXiv 2608.18215v1
• PDF
• Hugging Face

What people are saying

• LumiTokens: 3D Relighting via Token-Space Lighting Transformation qixing_huang

Related papers

• Staged Blender Code Beats SAM-3D Agents at Single-Image Inverse Graphics
• TAPe+ML claims 88.1% ImageNet and 65.3 COCO AP with under 100k parameters
• Shared hypernetwork synthesizes VideoQA prompts on demand, cutting NExT-QA forgetting to 4.76
• GPT-6 Astra as robot policy hits 22.48% on RoboDojo, beating 40 public VLAs
• RegVGGT keeps 1% of tokens per frame for streaming feed-forward reconstruction
• D3GS co-guides sparse-view 3DGS with metric depth, DINO, and RGB diffusion

All paper explainers