Patch-Based Generalized Neural Texture Upscaling
Shin Fujieda, Takahiro Harada
cs.GR
2026-11-05
A single generalized MLP upscales textures 2× at sampling time, with no per-asset training. Hangar diffuse LPIPS falls from bicubic 0.156 to 0.075, and PFT stays ahead.
High-resolution textures spend disk and VRAM at the same time. Offline super-resolution, from SRCNN through PFT, can sharpen missing detail without a per-asset fit. Those models consume a whole image and cannot serve the random access a texture fetch needs, so they have to run before the engine loads. The sharp texture still lands on disk and in VRAM.
NTC overfits one small MLP to a PBR set and decodes it at fetch time, cutting disk and VRAM together. Every asset needs its own compression pass, and every set carries different weights. A wavefront that touches several materials diverges. Matrix cores want a group of threads on one matrix, and that divergence fights the hardware. NTBC writes hardware block-compression blocks and leaves decoding to the texture unit. Quality stops at what that format can hold, and the asset pipeline still gains a step.
This method puts the upscale inside the sampler. One generalized MLP reads a low-resolution neighborhood and writes texels at twice the resolution. Texture files stay untouched.
The network never sees a full texture. Patch side n is 1, with a one-texel margin and three channels, so the input is 3×3 and the output is 2×2. The tiny patch keeps the fetch small. Without the margin, neighboring blocks fail to meet. Missing border texels are mirrored, which assumes a smooth edge.
Bilinear interpolation lifts the patch to the output grid first and carries the low frequencies. The MLP predicts only the residual, and the texel is the sum. Bilinear is already a usable approximation, so the network learns the high frequencies it drops. L1 loss came out sharper than MSE under the same setup. Training uses Adam at 1e-3, with optional cosine decay to 1e-6.
The 27-dimensional neighborhood is projected to width 256 and then through four residual blocks. Each block is two linear layers with a ReLU between them, a skip add, another ReLU, and a clamp at 1. The head emits 12 numbers, a 2×2 three-channel patch. That is small enough to live in the sampling shader. Every texture shares the weights, so a material change does not send threads through a different matrix.
Training data is Poly Haven: diffuse, normal, and ARM, about 560 maps each, long side at most 2048. Class-specific models often fit one class better. Hangar diffuse and normals follow that pattern. Museum normals are the exception: a normals-only model did worse. One model on all three classes is not the per-class winner, and it is the balanced choice on both test scenes, with a single weight set to ship. Every number below comes from that model. Hangar, Victorian, and Museum are held out. Poly Haven is split internally.
Baselines are bilinear, bicubic, and Lanczos, plus PFT as an offline quality reference. PFT is much larger and cannot run at sampling time. LPIPS is the main metric. Lower is better. PSNR and SSIM move less, and a blurrier image sometimes scores a higher PSNR than one that looks sharper.
| Dataset | Map | Ours | Bilinear | Bicubic | Lanczos | PFT |
| Poly Haven | diffuse | 0.239 | 0.396 | 0.369 | 0.383 | 0.224 |
| Hangar | diffuse | 0.075 | 0.193 | 0.156 | 0.157 | 0.043 |
| Hangar | normal | 0.023 | 0.088 | 0.067 | 0.065 | 0.014 |
| Hangar | ARM | 0.041 | 0.094 | 0.068 | 0.065 | 0.024 |
| Victorian | diffuse | 0.1026 | 0.2254 | 0.1895 | 0.1918 | 0.091 |
| Museum | diffuse | 0.052 | 0.142 | 0.107 | 0.107 | 0.048 |
Class-average LPIPS is lower than all three analytic filters. PFT is still ahead, and the gap depends on the set. Hangar diffuse is 0.075 against PFT at 0.043. Poly Haven diffuse is 0.239 against 0.224. Some Hangar and Victorian normals, and some Museum diffuse maps, match PFT on a single texture.
Pixel error does not agree. Hangar diffuse PSNR is 36.86 dB here, 35.42 dB for bicubic, and 39.38 dB for PFT. On Museum normals, bicubic reaches 40.23 dB and SSIM 0.970, against 38.44 dB and 0.966. LPIPS on that same row is 0.020 against 0.044. Pixel metrics and perceptual metrics rank the methods in opposite order.
In a HIP renderer, three Hangar cameras score FLIP 0.0519, 0.0415, and 0.0441. Bicubic scores 0.0573, 0.0441, and 0.0470. Bilinear scores 0.0626, 0.0473, and 0.0495. Close views show the gap in glossy reflection and in the diffuse maps. The numeric drop is about 0.005.
Overfitting one texture set usually helps. Hangar diffuse LPIPS moves from 0.076 to 0.059, and Victorian ARM from 0.047 to 0.033. Museum normals go from 0.020 to 0.021, which is worse.
The NTC comparison is one Poly Haven diffuse map. NTC uses PEPS with four frequencies, an 11-dimensional input, and an MLP of hidden size 64 with four layers. A 512×512×3 latent grid beats both this method and PFT, because those weights are fit to that exact image. This method roughly matches the 256×256×3 grid and loses on hard cases. The figure has no per-item numeric table. NTC also stores a network per texture, and that extra weight was left out of the comparison.
At hidden size 8, Hangar diffuse LPIPS is still 0.079, below bicubic at 0.156. The authors mark 128 as the knee. Removing the context margin moves LPIPS from 0.0755 to 0.0871 and PSNR from 36.87 dB to 33.79 dB. Turning the residual link off moves LPIPS from 0.0755 to 0.0796. A bicubic base instead of bilinear did not help.
A project that ships the low-resolution base can replace the sampler with this MLP and skip per-asset compression. One weight set means more materials do not mean divergent matrix code. The bar it clears is bilinear, bicubic, and Lanczos. It does not reach PFT's offline quality or NTC's compression ratio. A 2× side length stores about a quarter of the texels. The paper never measures the bytes.
There is a switch NTC does not have. After a diffuse bounce, where high frequency is unnecessary, the shader can read the low-resolution texel and skip the MLP. NTC has to decode its latent grid first. That switch is argued, not timed.
Runtime cost is unmeasured. Per-sample time, frame time, and the trade between width and throughput are left as future work. A clean place to hook the sampler does not show that a shader budget can hold the MLP. Each fetch reads 3×3 and writes 2×2. If neighboring samples do not reuse those four texels, the extra outputs are wasted. The paper cites wasted texels as the reason to stay at a 2×2 output, and it describes no cache.
The scale factor is only 2×. Detail finer than a source pixel is absent from the low-resolution map, and neither this method nor PFT restores it. Hard cases stay far apart. WeaponD on Hangar scores LPIPS 0.097 against PFT at 0.030. Denim on Poly Haven is 0.552 against 0.441.
The NTC comparison is a single texture, not a compression sweep. Nothing checks whether upscaled normals stay near unit length, or whether roughness and metalness stay in range. The loss is L1 on texels, not a radiance-domain physical constraint. Borders are mirrored under a smoothness assumption. Seams on maps that do not tile are not tested on their own.