TTB: Test-time Baking for Efficient Rendering of Decoder-only View Synthesis Models
Chin-Yang Lin, Ryo Hachiuma, Jaesung Choe, Min-Hung Chen, Yu-Chiang Frank Wang, Yen-Yu Lin, Wei-Chen Chiu, Yu-Lun Liu, Cheng Sun
NeurIPS 2026 (Spotlight)
cs.CV
2026-11-13
Test-Time Baking fits a fixed MLP to cross-view attention outputs. At 960p with 64 views it renders at 23 FPS against DVSM's 1 FPS, with PSNR 30.20 versus 31.11.
Decoder-only view synthesis skips explicit 3D. DVSM stores the keys and values of every cross-view attention layer in a KV-cache, encodes a scene once, and queries that cache for each new camera. Quality sits with the current front of the pack. Cost grows linearly with context tokens. On DL3DV at 960p with 64 input views, rendering falls to 1 FPS and prefilling takes 60 seconds. Dense inputs and high resolution are the regime these models are built for, and the linear query spends that advantage.
LaCT replaces cross-view attention with Test-Time Training: a fast-weight MLP updated for a few steps on the current scene. Rendering becomes constant-cost, and quality drops. Finetuning from a pretrained decoder-only checkpoint still leaves a clear gap. TTT fits raw value tokens, so the mixing attention already did across views is never stored.
Test-Time Baking keeps that MLP and points the inner loop at the output of cross-view attention. Attention mixes every context view first. The MLP memorizes the map from query location to that mixed output, and only the weights are kept. Each novel-view token is one forward pass, the same rendering complexity as TTT. Reconstruction still runs attention. The bill moves from every new view to once per scene.
The other fix is modality. Prefill tokens carry an image. Novel-view tokens carry only a camera. TTB splits each context view into a pair. The ground-truth token, RGB plus pose, supplies the attention target. The rendering token, pose only, is the MLP input, and it matches the novel-view distribution. The map takes a ray's latent coordinate to an appearance latent, a few steps of light-field fitting in latent space. The inner loop uses multi-step mean squared error. LaCT uses one unnormalized dot-product write.
The network has 12 blocks, width 768, and patch size 8, initialized from a DVSM checkpoint. Fast weights are a single-head SwiGLU MLP with hidden size 3072. Muon runs three inner steps. A linear head predicts a per-token learning rate around a base of 0.01. The target is a residual, easier to fit in a few steps than the absolute attention output. The baked MLP also writes back onto the image-carrying tokens, so later layers see one set of operators. When one MLP cannot hold more views, inference clusters cameras with k-means, about 16 views per group, and routes each novel view to the nearest group. One query stays constant-time.
On 140 held-out DL3DV scenes at 960p, TTB renders at a flat 23 FPS. TTB is 271M parameters and 12 blocks. DVSM is 163M. LaCT is 379M and 24 blocks.
| Method | 16-view PSNR / FPS | 64-view PSNR / FPS | 64-view prefill |
| LVSM | 22.18 / 20 | 21.96 / 20 | 59 s |
| LaCT | 24.70 / 10 | 28.30 / 10 | 7.3 s |
| DVSM | 26.86 / 4 | 31.11 / 1 | 60 s |
| TTB | 27.00 / 23 | 30.20 / 23 | 65 s |
| TTB, grouped | not reported | 31.33 / 23 | 65 s |
At 64 views TTB is 23 times faster than DVSM. PSNR is 30.20 against 31.11, and LPIPS is 0.131 against 0.104. Grouped baking reaches 31.33 PSNR and 0.108 LPIPS, slightly above DVSM, still at 23 FPS. Against LaCT at 28.30 PSNR and 10 FPS, that is about 2.3 times faster and 1.9 dB higher. At 16 views TTB's PSNR is 27.00, a bit above DVSM's 26.86. At 32 views it is 28.92 against 29.13. TTB--, which drops quadratic attention during reconstruction, scores 29.29 at 64 views with a 7.5 second prefill, next to LaCT's 7.3 seconds. Per-scene 3DGS takes more than 600 seconds and reaches only 26.43 PSNR at 64 views.
One or two novel views do not pay for the heavier prefill. After about 5 to 7 views, end-to-end time crosses over. At 64 views, 100 novel views take about 69 seconds against DVSM's 160.
Out of domain, still 64 views at 960p, TTB versus DVSM is 24.49 against 24.76 on MipNeRF360, 26.05 against 25.57 on Free, and 23.77 against 23.10 on Hike. Grouped baking is higher on all three. At a 512-pixel long side, TTB renders at 67 FPS against LagerNVS at 7 FPS. LagerNVS has 1052M parameters. MipNeRF360 PSNR there is 23.62 against 20.98. On two-view Re10K at 512 resolution, TTB scores 31.28 and DVSM 31.17. With only two views the speed gap shrinks.
Ablations use a smaller setup, so their frame rates should not be read against the 23 FPS headline. Swapping in TTT drops PSNR from 24.68 to 23.12 at 32 views. Full TTB returns to 24.84 at about 50 FPS. Residual targets and the write-back together add 0.52 PSNR. TTB trained from scratch already reaches 23.82, above pretrained TTT at 23.12. Starting from DVSM adds another 1.02. In the 720p scaling run, a single MLP flattens near 25.8 PSNR past 64 views. Grouped baking reaches 27.74 at 256 views, in line with DVSM's 27.56, at about 68 FPS against 12.6. Fast weights are a fixed 324 MB in FP32, about 85 times smaller than the KV-cache at 64 views and about 340 times smaller at 256.
The step is incremental. The payoff is deploy-time cost. Quality stays with the decoder-only model, and queries no longer slow down as views are added. For an interactive viewer or a camera path, the clock favors TTB after roughly 5 to 7 novel views. Memory is the sharper gap. Linear extrapolation in the paper puts DVSM's cache past 80 GB at 256 views and 544 by 960. TTB's scene state does not grow with the view count.
3DGS and LongLRM commit to explicit geometry, which has trouble with strong mirrors and translucent surfaces. At 64 views LongLRM's PSNR is 24.77 and TTB's is 30.20. LongLRM's table lists a higher frame rate because a Gaussian rasterizer is doing the drawing, a different clock from a network forward. Keeping attention inside the one-time prefill, and training an MLP to store its output, is a pattern that could move to long-context caches and to temporal attention in video models. The paper only draws that parallel.
The MLP has fixed capacity and saturates as views pile up. Grouping is inference-time k-means, about 16 views per group. The router is not learned, and groups are not blended. On the 720p scaling table, grouped reconstruction at 256 views takes 52 seconds, against 20.5 seconds for a single MLP. The main table lists the same 65 second prefill for grouped and single TTB at 64 views, so those two reports do not line up. Cross-view attention inside reconstruction is still quadratic in the view count. A shot that needs one or two images is a poor trade, and the end-to-end timings agree. The task is reconstruction: interpolate among observed views, with no claim on unseen directions.
Headline numbers come from a DVSM finetune. LaCT and LVSM trained from scratch are a different starting line. Scratch TTB still beats pretrained TTT, so initialization is not the whole gain, and there is no scratch DVSM control. The 23 FPS figure is 960p with no batching. The 720p ablation sits near 50 FPS. Frame rates do not compare across tables. The out-of-domain set is 22 scenes, and Hike contributes 6 of the stabler ones. Thin structures look cleaner than LaCT in the figures. There is no geometry error and no user study. A close PSNR does not mean every novel view holds up.