EDGAR: Efficient Diffusion-Driven Gaussian Avatar Refinement via Synthetic Failure
Kwan Yun, Minhyeok Bang, Colette Suhjung Koo, Junyong Noh
cs.GR, cs.CV, cs.LG, cs.MM
2026-11-05
EDGAR trains one-step diffusion on avatar failures and bakes the fix into the 3D model. Eight-video RGBAvatar PSNR rises from 20.51 to 22.99, with no diffusion at playback.
A Gaussian head avatar fit to one video or a few photos still has to invent geometry and texture for angles the camera never saw. Side views, a wide-open mouth, or raised brows then smear, stretch, or leave stray Gaussians around the hair. Avatars from a dense camera rig rarely show those traces.
SDEdit cleans a render by noising and denoising it with a pretrained diffusion model. High-frequency detail can return. Nothing in that pass is told who the person is, so identity drifts on unfamiliar poses, and the random noise makes neighboring frames disagree. An animatable head also has to keep the expression, and playback cannot afford a diffusion pass on every frame.
EDGAR, from KAIST, learns a narrow map: a degraded render plus one photo of the same person should land close to the true frame at that view. No such pairs existed, so the training set is the triplet collection EDGAR-SET, with a failed render, a ground-truth frame, and a reference frame.
The real half is NeRSemble reconstructed with RGBAvatar: 99 identities, deliberately trained on only 1 to 4 views. The synthetic half is CAFCA reconstructed with the original 3D Gaussian Splatting: 1,500 identities and at least 5 views, which supplies harsher angles and expressions. The reference is a ground-truth frame of the same person at another view or expression. At test time it is a frame from the monocular video that trained the avatar.
Pairs with a high LPIPS distance, or with expression and head pose already misaligned, are removed. About half the pairs go. What remains is about 400K images at 512 resolution. View counts of one, two to four, and five or more make up 23%, 51%, and 26% of the kept set. Severe structural failures are outside this distribution.
The backbone is SD-Turbo, a one-step distillation of Stable Diffusion. Residual blocks, timestep injection, and text cross-attention stay frozen. Self-attention is updated, and the decoder gets a LoRA. Added layers start at zero, so training begins identical to the pretrained model. The prompt is fixed as “High resolution portrait.” and the timestep is 250.
The reference goes through frozen Sapiens. A fusion block turns those features into the keys and values of cross-attention. Degradations range from heavy to nearly clean, so a router picks one of six expert MLPs, with a load-balancing term against expert collapse.
The image loss is L2, LPIPS, and a VGG Gram term, each weighted 1. Expression uses intermediate features from POSTER V2 with an L2 weight of 8, aimed at lips and eyes. Load balancing is weighted 0.001. Training runs 40,000 steps at batch size 4 with gradient accumulation of 2, learning rate 2×10⁻⁵, about 40 hours on one A6000.
The 2D fixes are then written into the Gaussians. Fifty views are rendered, passed through EDGAR, and used as targets for 500 steps with L1 weight 0.6 and SSIM weight 0.4. One reference is sampled and kept fixed. No Gaussians are added, and playback never calls EDGAR.
The matrix is CAP4D at 1, 4, and 8 images, RGBAvatar at 1, 4, and 8 videos, and GaussianAvatars at 8, 10, and 12 videos. Self-reenactment uses 10 unseen NeRSemble identities. Cross-identity reenactment uses 6 INSTA identities. Every score is on novel views. SDEdit is the same SD-Turbo at the same timestep, with no reference and no 3D write-back.
On RGBAvatar and GaussianAvatars, EDGAR leads CLIP similarity, L1, LPIPS, and PSNR at every input count. On CAP4D it has the best CLIP and the lowest L1, but those L1 gaps sit in the fourth decimal place. LPIPS stays lower for the raw model, and PSNR stays higher for SDEdit.
| Setting | Metric | Raw | SDEdit | EDGAR |
| RGBAvatar, 8 videos | PSNR | 20.5128 | 20.5109 | 22.9937 |
| RGBAvatar, 8 videos | L1 | 0.0502 | 0.0555 | 0.0332 |
| RGBAvatar, 8 videos | LPIPS | 0.2100 | 0.2752 | 0.1894 |
| RGBAvatar, 4 videos | PSNR | 19.4950 | 19.9181 | 21.5199 |
| GaussianAvatars, 10 videos | LPIPS | 0.2661 | 0.2377 | 0.2050 |
| GaussianAvatars, 10 videos | PSNR | 18.5994 | 18.9219 | 19.3206 |
| CAP4D, 1 image | CLIP | 0.9369 | 0.9078 | 0.9449 |
| CAP4D, 1 image | LPIPS | 0.2133 | 0.2728 | 0.2147 |
| CAP4D, 8 images | PSNR | 18.4773 | 18.6645 | 18.5329 |
On 8-video RGBAvatar, PSNR moves from 20.5128 to 22.9937 and L1 from 0.0502 to 0.0332. On 10-video GaussianAvatars, LPIPS falls from 0.2661 to 0.2050, while PSNR only moves from 18.5994 to 19.3206.
Cross-identity CSIM at 1, 4, and 8 views is 0.5510, 0.5642, 0.5651 for the raw avatar, 0.5169, 0.5326, 0.5419 for EDGAR, and 0.3178, 0.3432, 0.3380 for SDEdit. EDGAR sits about 0.02 to 0.03 under the avatar it started from. The paper calls that gap negligible. The original score is not preserved, though the drop is far smaller than SDEdit’s.
Foreground temporal warping error, motion-compensated RGB L1 times 10⁻³, averages 4.704 before refinement and 4.403 after. CAP4D goes from 1.759 to 1.696, RGBAvatar from 7.424 to 5.574, and GaussianAvatars from 4.929 up to 5.938. The pipeline with the cleaner spatial numbers gets less stable in time.
Where renderer depth is calibrated, CAP4D and GaussianAvatars improve both relative depth error and reprojection coverage at every input count. At 10 videos, depth error moves from 0.0072 to 0.0064 and coverage from 0.5112 to 0.5261. RGBAvatar depth comes from VGGT on the renders. It improves at 4 and 8 videos and gets worse at 1 video: depth error 0.0083 to 0.0097, coverage 0.4855 to 0.4261.
For yaw of at least 30° and a lip gap of at least 0.10 times mouth width, 4-video RGBAvatar cuts foreground L1 from 0.08785 to 0.06112 and LPIPS from 0.17563 to 0.14150. Ten-video GaussianAvatars improves on both. Four-image CAP4D gets worse on both, L1 0.07475 to 0.08084 and LPIPS 0.23054 to 0.23507.
On an RTX 3090 the full refinement averages 5.04±0.11, 7.96±0.98, and 16.60±5.98 minutes for CAP4D, RGBAvatar, and GaussianAvatars, including load, pseudo-targets, 500 steps, and the checkpoint. Base optimization is about 484, 106, and 239 minutes, excluding preprocessing. After refinement, frame rate stays within 2% of the base model: 50.3 vs 49.4, 813.8 vs 826.3, and 153.4 vs 156.1 FPS.
Ablations on CAP4D show the reference image matters most. At one view, dropping it cuts CLIP from 0.9449 to 0.9163, raises LPIPS from 0.2147 to 0.2683, and cuts CSIM from 0.5169 to 0.4508. Dropping the synthetic split or the expression loss also cuts CSIM. Six experts versus no MoE moves most entries only in the third decimal place.
Fifteen participants, mean age 27.6, each saw 20 cross-identity video pairs, all against CAP4D. Versus the raw avatar, preferences were 87.33% for animation, 91.33% for identity, and 96.67% overall. Versus CAP4D plus SDEdit, the figures were 94.67%, 98.0%, and 98.0%. On CAP4D, people and the LPIPS and PSNR numbers disagree.
Playback never runs diffusion, so the frame rate is the original avatar’s. A finished model needs a few minutes to a quarter hour of post-processing, not a rebuild.
The gains sit on video reconstruction, and they are largest for RGBAvatar. On CAP4D, CLIP and the user study move toward EDGAR while PSNR and LPIPS often do not. The usable range is appearance error that is still aligned to the person.
Against SDEdit on the same SD-Turbo backbone, the reference image is what holds identity, and the multi-view targets are what put the fix into one shared Gaussian avatar.
EDGAR-SET drops pairs with large LPIPS gaps. Severe structural mistakes and regions that were never observed remain after refinement. Mouth interiors and translucent surfaces stay hard. Pseudo-targets are refined per view, so they can contradict one another, and one Gaussian set can only average the conflict.
GaussianAvatars records a higher temporal error, and the paper does not show the frames where that shows up. RGBAvatar was used to build the real half of the training set and is also a test baseline. The identities are new. The failure pattern of that pipeline is not clearly new. The user study covers only CAP4D, with 15 people, and skips the video setting where the numbers move most. The reference is sampled once and then frozen, with no test of a different frame. The external control stops at SDEdit on the same backbone.