Examining Joint Demosaicing and Denoising for Single-, Quad-, and Nona-Bayer Patterns
SaiKiran Tedla, Abhijith Punnappurath, Luxi Zhao, Michael S. Brown
eess.IV
2025-04-09
A one-hot pattern embedding lets a single demosaicing model beat specialized Quad/Nona networks and match Single-Bayer, while being smaller and faster.
Camera sensors are covered by a color filter array (CFA), traditionally in the Bayer pattern, and demosaicing reconstructs a full RGB image from the single-channel mosaic. Modern smartphones often carry several sensors with different patterns: Single-Bayer (1x1), Quad-Bayer (2x2), and Nona-Bayer (3x3). The strongest demosaicing today is done by deep networks that jointly denoise, and the standard practice is to train one model per pattern. That costs memory, and when a user zooms and the ISP must switch sensors, models must be preloaded on the NPU or swapped in, adding latency. The authors want a single model that handles all three patterns.
They first confirm that JDNDM, a strong RCAN-based Single-Bayer model, trained on shuffled Quad and Nona data actually beats the specialized BJDD and SAGAN, which suggests dedicated networks are not strictly necessary. Using JDNDM as the backbone, they try three unified designs: a remosaic-based multi-head model (SRUM), a shared-latent-space multi-head model (LSUM), and the winning embedding-based model (ESUM).
ESUM is straightforward: the input grows from a single-channel mosaic to four channels, the first being raw mosaic intensity and the other three a one-hot encoding (red, green, blue) of each pixel's color filter, so the network explicitly knows which filter produced each pixel and which pattern is in play. The packing convolution from JDNDM is removed. They also introduce a mosaic maskout augmentation: at training time, 0 to 1% or 0 to 5% of pixels are randomly dropped and the one-hot mask is updated accordingly, which improves performance and doubles as dead-pixel correction, since sensors have about 1% dead pixels that are normally interpolated separately before demosaicing. For evaluation they capture a Hard Demosaicing Dataset (HDD) of 638 RAW images across 17 scenes built for high-frequency detail, since the existing Pixelshift200 has only 200 images and lacks hard detail.
For anyone working on mobile imaging or ISPs, this is a practical engineering lesson: rather than training a separate network for each new sensor pattern, feed a unified network a lightweight embedding of the pattern and save memory, cut switching overhead, and gain accuracy. Dead-pixel correction coming along for free is a concrete bonus. It is also a clean example of a simple embedding replacing a specialized multi-head structure.
This is a fairly specialized task, of interest mainly to computational photography and mobile ISP engineers, and far from general AI. The unified model is still built on the specific JDNDM backbone, and whether it transfers to other backbones is unverified. Dead-pixel ratios and noise models rely on sensor-specific calibration, and cross-device robustness is not thoroughly discussed. The conclusions rest on the self-captured HDD and Pixelshift200, so end-to-end performance on real phone output needs further validation.