Thinking Outside the Box: Can Language Models Rely on External Guidance Selectively?
Minghan Wang, Boyuan Wang, Jinhang Zuo, Yuxin Tao, Fang kong
cs.CL
2026-09-30
Bad guidance costs frontier models up to 43 points on Box2-Bench; training Qwen on bad workflows alone cuts that damage from 20 to 7, and RL restores good-workflow gains.
Agent harnesses, the scaffolding around a model that organizes planning, feedback, and tool use, usually encode human-designed workflows: ordered step goals the model follows. The whole arrangement assumes the human designer is more reliable than the executor. As models get stronger that assumption breaks. A capable model may find a better route than the prescribed workflow and get penalized for following it; strip the workflow away and genuinely useful guidance is lost. Harness-executor mismatch is common in deployment, so this is a real problem, not a hypothetical one.
The paper calls the missing skill "thinking outside the box": follow guidance when it helps, override it when it misleads. It argues this is a dimension raw task scores cannot see, since a model that aces a benchmark can still follow an outdated SOP all the way to a wrong answer. Existing agent benchmarks measure task completion or procedural compliance, not how reliance shifts as guidance reliability changes.
Box2-Bench is a controlled comparison. Task, environment, evaluator, and model stay fixed; only the workflow varies across five conditions: none, good throughout, bad throughout, good-then-stops (Partial), and good-then-bad (Mixed). Partial and Mixed share the same useful prefix and switch point, so their difference isolates continuing with misleading guidance versus receiving nothing further. Four paired effects follow: utilization, robustness, and two recovery measures for guidance that stops or turns misleading.
Workflows are generated in bulk by a fixed external model and audited by a second one. Bad workflows follow a preparation-error-propagation structure: four low-information setup steps, one concrete wrong decision at step five, then three steps that carry the error through. Approved workflows are frozen and shown as identical strings to every evaluated model.
Training uses only bad workflows; good ones are held out for evaluation. Counterfactual SFT samples the base model for a verified correct solution, has a planner synthesize paired good and bad workflows, and trains on (problem, bad workflow) leading to the correct solution. One property matters here: this objective constrains nothing about behavior under good workflows, so blanket distrust and selective override fit the data equally well, and preserving utilization must come from generalization. A second stage runs outcome-based RL with GRPO-style group-relative advantages and clipping; reward is task success only, binary answer correctness on math and environment return on WebShop, never agreement with the workflow.
| Setting | Metric | Result |
| Frontier models, 12 pairs | Good workflow | improves 8 of 12; GLM 5.2 gains +16.8 on AutomationBench |
| Frontier models, 12 pairs | Bad workflow | hurts all 12, by 2.3 to 43.3 points; Gemini 3.7 Flash falls from 76.7% to 33.3% on OpenR1-Math |
| Frontier models, 12 pairs | Mixed vs Partial | lower in 10 of 12; trust built by a useful prefix carries over |
| Qwen3-4B, AIME 2026 | Robustness | damage shrinks from -20.0 to -6.7 after SFT |
| Qwen3-4B, AIME 2026 | Utilization | +13.3 base, -3.3 after SFT, +6.7 after RL |
| Qwen3.5-9B, WebShop | Robustness | damage shrinks from -9.8 to -0.6 after SFT |
SFT alone also destroys the benefit of good workflows, which reads as blanket distrust rather than selective reliance; RL restores utilization while keeping most of the robustness gain. The Mixed-Partial gap on AIME moves from -14.4 to +1.1, so the model can let go when guidance turns bad mid-way. Scaling does not fix this by itself: bad workflows still hurt five of six open-weight pairs, costing Qwen3-8B 26.7 points on AIME and Qwen3.5-27B 11.4 points on WebShop.
Transfer tests run the same checkpoints with no further training. In the multi-agent Economy of Minds environment, the SFT model raises episode success from 18.0% to 22.7% and repairs 11 wrong peer answers while corrupting only 1, against 2 and 2 for the base. On LongMemEval memory questions, corrupted-memory scores rise from 65.6% to 74.4% while reliable-memory accuracy stays in the 92-95% range, and every correct answer under corrupted memory came from a trajectory that checked the archive first.
For teams building agent platforms, this adds a directly usable check beyond task scores: shipped workflows will not stay correct forever, and the executor's selective reliance is the last line of defense. The training recipe is cheap, needing only bad workflows and a success signal, no good-workflow labels. Be clear about the evidence split: the benchmark findings on frontier models are solid, while the training results rest on 4B/9B models and roughly a hundred RL prompts, closer to a proof of concept.
The authors list two: workflows come from a fixed external model rather than humans with diverse intentions and error patterns, and only frozen workflows are tested, leaving interactive, ambiguous, or partially correct guidance to future work. The audit is model-based adjudication, and the paper explicitly reports no inter-annotator reliability coefficient.
Three more concerns from a close read. Scale is small: 30 task bundles per frontier domain, 30 AIME problems, RL pools of 111 math prompts and 153 WebShop tasks, and the memory transfer uses 30 questions with 3 rollouts, so directions are credible but magnitudes may not be. The deeper tension goes undiscussed: models are trained to follow explicit instructions, so a willingness to override a written procedure is a capability in this benchmark but a potential defect in production, where a user may genuinely require an SOP to be followed as written.