GGSS: Geodesic-Gated Spherical Steering for Inference-Time Debiasing of Generative Vision-Language Models
Yiqun Sun, Junyu Chen, Pengfei Wei, Lawrence B. Hsieh
EMNLP 2026
cs.CY, cs.CL, cs.CV
2026-08-26
GGSS geodesically steers visual tokens on frozen generative VLMs, posting the lowest average bias on all four models with MMStar within 0.6 points of baseline.
Generative vision-language models now show up in hiring and decision-support pipelines. Two photographs that differ only in perceived race or gender can still produce systematically different salary, education, or occupation answers. Inference-time debiasers from the CLIP era, INLP, LEACE, BendVLM, operate on a single global embedding with a hard projection or a mean shift. A generative VLM encodes an image as many visual tokens that then enter a decoder-only language model. Porting those operators onto the vision-to-language projection layer is often worse than leaving the model alone.
Three failures show up. Euclidean subtraction changes both direction and norm, and the damage grows with scale. Demographic signal lives in a minority of tokens such as faces and clothing; background and pose tokens carry almost none, so a uniform projection over-corrects. Race-like attributes are multi-class, so a single bias direction is the wrong object.
GGSS is a two-stage, frozen-model hook.
Offline, it uses 480 real, pixel-aligned face photographs from REFLECT/FOCUS: 6 occupations, 8 identities, 5 perceived races, 2 genders. Visual tokens are pooled, normalized onto the unit sphere, and summarized by a spherical Fréchet mean within each fixed context. Attribute-induced tangent shifts are stacked and factored by SVD; the top k = |A| - 1 right singular vectors become the bias subspace. The median and standard deviation of bias-coordinate magnitudes on the discovery set later calibrate the gate. Discovery occupations are held out from evaluation.
At inference, each visual token keeps its original radius, maps into the tangent space at a global spherical reference, has its bias coordinates zeroed, and maps back to a target direction. A sigmoid gate compares that token's bias magnitude with the discovery distribution: high-signal tokens rotate farther, low-signal tokens barely move. Spherical linear interpolation (Slerp) then walks the geodesic from the original direction toward the target, and the original radius is restored. Norm preservation is a mechanical guarantee, stated as Proposition 1.
Default gate: kappa = 5, floor 0.3. Steering strength alpha is chosen from {0.25, 0.5, 0.75, 1.0, 1.5} by the same best-avg-alpha rule for every method and model.
Backbones: Pixtral-12B, LLaVA-1.6-Vicuna-7B, LLaVA-1.6-Mistral-7B, Qwen3-VL-4B. Three probes: race multiple-choice salary/education scored by Jensen-Shannon divergence, race two-alternative forced choice scored by standard deviation, and the nurse/doctor gender gap. Baselines are INLP, MeanDiff, BendVLM, and LEACE, all hooked at the same projection layer, plus a fairness instruction with no steering.
| Model | GGSS mean bias drop | Strongest baseline |
| Pixtral-12B | -55% | BendVLM Euclidean -37% |
| LLaVA-Vicuna-7B | -90% | INLP spherical -86% |
| LLaVA-Mistral-7B | -80% | INLP spherical -78% |
| Qwen3-VL-4B | -60% | INLP spherical -49% |
Peak per-task drops: Vicuna nurse/doctor 0.600 to 0.025 (-96%), Vicuna MCQ JSD 8.70 to 1.36 (-84%), Pixtral 2AFC 0.243 to 0.096 (-61%). Four-model mean -71.4, worst case -55.3, versus -63.5 / -30.8 for LEACE plus the same gate. Paired permutation tests are significant on three of four backbones; Vicuna combined p = 0.068, nurse/doctor alone p < 10^{-4}.
MMStar at the same operating point stays within +/- 0.6 points of the unsteered model; all eight McNemar tests are indistinguishable from baseline. INLP spherical knocks 6.1 points off Pixtral. Prompt-only fairness is unstable and can raise Vicuna MCQ bias by 85%.
Ablation on Qwen3 MCQ at alpha = 1.0: unsteered JSD 14.646; no gate 5.612 (-61.7%); hard projection plus gate 4.533 (-69.1%); full GGSS 2.414 (-83.5%). Spherical geometry only clearly wins at 12B: Euclidean forms significantly hurt Pixtral MMStar (-6.3 points), while geodesic forms keep it. Over-steering produces a 9x overshoot and a 72/80 unparseable cell in Euclidean; the geodesic never did.
This is a hook for a frozen checkpoint. No retraining, no weight edits. Alpha is a dial: on Qwen3 at 0.5, race recognition stays within 4 points of baseline while 55% of the MCQ reduction is already in. Tasks that need the attribute can use a milder alpha or turn the gate off. Code is public.
The contribution is geometric and selective, not a fairness certificate. Under one shared operating-point protocol, GGSS is the most reliable of the four-model suite.
The study covers perceived race and binary gender, four models, and three probes. Intersectional groups, multilingual prompts, and open-ended generation are untested. Alpha still has to be chosen from held-out bias numbers; there is no tuning-free rule. At high strength the intervention shades into attribute removal, so any task that must report demographics needs a different operating point. Labels are curator-assigned perceived categories, not self-identification. The same spherical rotation can amplify bias as easily as reduce it; the ethics statement says so.
Identities are held out from discovery, but alpha is still selected on the probe suite, so absolute reductions may be optimistic. Cross-fitted folds still rank GGSS near the top, matching the paper's alpha on six of eight folds.