Beyond Correctness: Benchmarking and Aligning Response Behaviors in Hybrid-Thinking MLLMs
Xinming Wang, Weinong Wang, Hongming Yang, Yansong Lin, Zheng Ruan, Shangpin Peng, Qiming Peng, Nan Qiao, Fengyuan Lu, Guoqing Ma, Marito Li, Songyang Zhang, Saiyong Yang, Han Hu, Yonglong Tian, Xu-Yao Zhang
cs.CV
2026-08-13
2,415 multimodal prompts show every hybrid-thinking MLLM fails more in non-thinking mode, max gap 48.64 points; PatternRL cuts 8B trigger 14.35 points vs correctness-only RL.
Hybrid-thinking multimodal models expose two knobs on one set of weights: a thinking mode that spends extra test-time compute, and a non-thinking mode that answers under a tight latency budget. That split is convenient in a product. The text the user actually sees should still meet one bar: no leaked scratch work, no loops, no unresolved contradictions, no fake reasoning wrapper. Accuracy training does not police those properties.
CASIA and Tencent call the missing requirement response-pattern alignment. They run 25 open and closed configs, dense and MoE, on the same image-prompt set in both modes. Non-thinking trigger rates are higher in every pair. The largest gap is 48.64 points, on Kimi-K2.6. Stronger task scores do not close the interface gap.
PatternEval is a failure-enriched diagnostic set, not a prevalence survey. From visual perception and grounding, OCR and structured images, and multimodal knowledge reasoning, they keep 2,415 image-prompt pairs across nine categories, biased toward cases that surface bad replies. Only the user-facing answer is scored. A separate reasoning channel, if the API exposes one, is ignored.
Four labels:
Qwen3-Max grades correctness with each source dataset's rules. Seed-2.0-Pro, with the image, assigns the four binary labels. Any positive label counts as a Trigger. Labels can co-occur, with CoT-priority attribution so leaked deliberation is not double-counted as contradiction or performative reasoning. Judges were calibrated on 2,500 human-labeled rollouts. CoT and repetition have surface cues and score well; contradiction and performative reasoning need cross-sentence consistency and visual grounding, and their F1 is much lower.
PatternRM is Qwen3.5-27B fine-tuned on about 52k responses where three teacher judges agreed on the full four-label vector. It reads text only and predicts the four bits directly, at 71.3% macro-F1. PatternRL then runs GRPO on about 44.2k multimodal math, logic, and document prompts. The main reward is boxed-answer correctness. With probability 0.6, PatternRM adds a penalty: 0.05 each for CoT and repetition, 0.02 each for contradiction and performative reasoning, capped at 0.1. Wrong answers stay at 0; correct ones land in [0.9, 1]. BaseRL uses the same data and verifier without the penalty. The 4B policy trains for 600 steps, the 8B for 400.
All 25 pairs have a positive pattern gap. Seventeen exceed 20 points. CoT leakage dominates. Marginal trigger rates across 50 model-mode rows are 12.75% CoT, 8.49% repetition, 4.72% performative reasoning, and 2.74% contradiction. Because of CoT-priority attribution, those are not independent prevalences.
| Model | NT Trigger | Think Trigger | Δpat |
| Kimi-K2.6 | 49.63% | 0.99% | 48.64 |
| Qwen3.5-397B | 29.32% | 4.23% | 25.09 |
| Seed-2.0-Pro | 6.94% | 3.13% | 3.81 |
| GPT-5.5 | 5.04% | 0.78% | 4.26 |
Inside Qwen3.5, thinking Trigger falls from 22.19% at 4B to 4.23% at 397B. Non-thinking Trigger drops through 27B, then sits between 28% and 33%. The cross-mode gap does not shrink with scale. Longer replies fail more often (Pearson r=0.64 non-thinking, 0.84 thinking). In the longest sextile, even correct non-thinking replies still trigger at about 56%.
| Setting | Acc | Trigger |
| Qwen3-VL-8B non-thinking | 49.37% | 36.77% |
| BaseRL | 50.57% | 44.33% |
| PatternRL | 50.71% | 29.98% |
Correctness-only BaseRL makes the bad patterns worse. Versus BaseRL, PatternRL cuts non-thinking Trigger by 13.08 points on 4B and 14.35 on 8B, with PatternEval accuracy moving by less than one point. On ten general benchmarks, 8B overall accuracy slips from 79.59 to 78.89; 4B slips from 77.89 to 76.02. The smaller model pays more.
If you ship a hybrid-thinking switch and only watch accuracy, you will undercount product failures on the fast path. Leaked scratchpads, loops, and fake reasoning are visible to users. PatternEval is a stress set, so the rates are not production frequencies. What it does show is that scale and accuracy leave this gap open. PatternRL is a drop-in GRPO penalty; 8B holds accuracy and form together better than 4B.
PatternRL does not erase the failures. 8B non-thinking Trigger is still 29.98%, 14 points above thinking-mode 15.98%. The paper says the habits may already be baked in at SFT, and a light RL penalty is too late. The 4B model loses more on math, which is consistent with small models using verbose, self-correcting traces that the penalty then forbids. PatternRM does not see the image, so visually grounded contradictions can slip through, and the judge F1 on those labels was already weak. The benchmark measures conditional robustness, not the failure rate you should expect in traffic.