EXPL-FR: Explaining Face Recognition Models via Vision-Language Alignment
Guray Ozgur, Mustafa Efe Tamyapar, Naser Damer, Fadi Boutros
ECCV 2026 Workshops
cs.CV
2026-08-21
An image-only adapter maps CLIP into frozen face-recognition space so 978 text prompts become audit axes, ranking models by RFW ethnicity errors with no labels.
Face recognizers sit near ceiling accuracy and still return a cosine with no vocabulary. Spatial heatmaps say where a matcher looked. Attribute studies that need CelebA's 40 labels, or a controllable generator, cannot cheaply name a new factor. A deployed FR encoder has no decision head, so concept-bottleneck methods that read a classifier do not apply. EXPL-FR, from Fraunhofer IGD, asks a narrower question: can a frozen matcher be read in its own embedding space with ordinary language.
Both sides stay frozen: a VLM image/text pair (CLIP ViT-B/16 by default, SigLIP as a second) and a target FR encoder (AdaFace ViT-B/WebFace4M as the primary, plus ViT-S and two ResNet-100s). The only trained piece is a four-layer MLP adapter of about 1.05M parameters. It is fit on WebFace4M with a cosine loss that pushes VLM image embeddings toward FR embeddings, and it never sees text or attribute labels. Because CLIP's encoders share one space, the same adapter is applied to text, turning 978 prompts in 22 categories into FR-space anchors.
Not every concept survives. An FR model buys invariance by discarding pose, lighting, and scene. A label-free detectability score uses VLM pseudo-labels to measure separability in VLM space versus adapter-mapped FR space; the drop is the discount. The top 100 prompts by AUCF become the semantic signature. Explanations come at three grains: one image, an identity aggregate, and a genuine/imposter/morph difference. Audits run at three supervision levels: human labels as an upper bound, VLM pseudo-labeled groups, and a prompt-only axis (PCA-1 of adapter-mapped prompts). A new attribute costs a written sentence.
On the primary model, unaligned cross-encoder verification is 50.29%. Mixing gθ(ϕv) with FR after alignment reaches 94.56%, 2.88 points under FR self-verification at 97.44%. CLIP's own identity verification is 82.33%; the adapter lifts it to 92.58%. The 978 prompts carry almost no identity in CLIP space (51.98%) and 71.66% after the adapter; that +19.68 isolates a cross-modal transfer that was never trained on text. TinyFace is the break: mixed rank-1 is 13.49% against FR's 73.77%, because extreme low resolution kills the VLM image encoder. The deployed signature pairs FR embeddings with text anchors and does not take that path.
Detectability keeps eyewear at 0.955 and hair color at 0.947, and discounts distance (0.853), scene, camera angle, face coverings, and lighting. The top-100 signature separates identities better than the full 978-d vocabulary (0.931 vs 0.910 on the primary). On CelebA, label-free dependence ranks attributes with Spearman ρ=0.64 against the labeled KS statistic.
On RFW, all three supervision settings rank four models by per-ethnicity error at mean Kendall τ=0.92, and ranking by mean dependence matches ranking by mean real error. On GAN-Control, prompt-only sensitivity tracks real EER at ρ=0.95 (labeled 0.90, VLM-proxy 0.83). Eight-way attribution falls off with less supervision: labeled 82-90%, proxy 33-34%, prompt-only 16-19%, chance 12.5%. Pose and illumination are VLM blind spots on aligned crops; the audit still emits numbers that look like findings and should be marked inconclusive.
The readout is post-hoc and black-box. The deployed encoder is not retrained, and a new attribute does not need new labels. Models can be ranked by RFW ethnicity error from prompts and unlabeled faces, which is more useful for selection than another LFW tick. The same readout shows which changeable cues the matcher uses: hair color and glasses are stable in web-scraped photos, so the model keeps them, and recoloring hair moves the embedding by 0.07-0.09, more than photometric factors it has erased. That dependence is a real weakness. The audit is doing its job when it surfaces it.
Vocabulary projection remains 25.78 points below the FR upper bound, some identity structure is not nameable, and prompts are not disentangled, so readings are directional. Detectability is scored against the VLM's own pseudo-labels, so concepts CLIP cannot rank (pose, illumination, tiny resolution) can leak through or go missing. The adapter flatters AUCF; the same detector on true FR embeddings averages 0.740. Prompt-only attribution is near chance, so the method can rank models and costly variations, not pinpoint which attribute changed. CLIP saw faces in pretraining. The claim is no attribute labels in the adapter, not a face-free stack.