Reading Router Logits: Lightweight Multimodal Safety Detection for MoE VLMs

@shangbinfeng posted a five-part thread on the team's arXiv paper "Reading, Not Manipulating: Leveraging Router Logits for Multimodal Safety in MoE Vision-Language Models." The work targets "compositional risk" in multimodal models — where a harmless image plus a harmless question can combine into a harmful request — and proposes an extremely lightweight detection method: during prefill, read the router logits of the final prompt token across all MoE layers, feed them into a 6K-parameter logistic regression to get p(unsafe), and reject if it exceeds a threshold. No extra forward computation, no weight changes, no touching the routing.

Confirmed

Existing defenses fall short: prompts, SFT, and expert-manipulation approaches are either inconsistent across models or trade excessive refusal for safety — image safety prompts caused a 56.5% false refusal rate on Qwen3-VL
Results: HoliSafe's overall error rate dropped significantly from 33.6%, with reductions up to 92%; OOD attacks dropped to zero
Key finding: manipulable experts correlate with detector features at only r≈0.03 — safety information is distributed and redundant across the MoE, so any layer group works, and even random expert subsets achieve near-equal performance — which is why "reading out" beats "manipulating"
Limitation: the signal partially depends on compositional specificity; removing unsafe-image data from training sent ASR soaring from 14.5%

Why it matters

The method gives MoE VLMs near-zero-cost multimodal safety interception before generation, avoids the over-refission common in traditional safety alignment, and reveals interpretable safety structure in MoE routing states.

2026-10-24 ~ 2026-10-24 · 5 related posts

Primary sources

Reading, Not Manipulating: router logits as a safety detector for MoE VLMs (arXiv) — shangbinfeng · 2026-10-24 03:19 (UTC+08:00)
MoE router logits are a free predictive safety signal for vision-language models — shangbinfeng · 2026-10-24 03:19 (UTC+08:00)
~6K-param router-logit detector cuts VLM safety errors by up to 92% — shangbinfeng · 2026-10-24 03:19 (UTC+08:00)

[source] MoE router logits are a free predictive safety signal for vision-language models — shangbinfeng · 2026-10-24
Benign image + benign question can be harmful: prompting, SFT, steering all fall short — shangbinfeng · 2026-10-24
[source] ~6K-param router-logit detector cuts VLM safety errors by up to 92% — shangbinfeng · 2026-10-24
Reading beats steering: safety info is distributed in MoE, detector uses ~6K params — shangbinfeng · 2026-10-24
[source] Reading, Not Manipulating: router logits as a safety detector for MoE VLMs (arXiv) — shangbinfeng · 2026-10-24