Ask Before You Optimize: Dynamic Pre-Formulation Clarification for Interactive Optimization
Sihan Ge, Yichen Lin, Chenyu Zhou, Jianghao Lin, Tao Yao, Dongdong Ge
math.OC, cs.AI
2026-09-04
InterOPT hits 67.5% Core Exact on OR-Clarify choice vs 50.6% for MC-D; 40.6% of free-form runs still stop too early.
LLMs can already turn a natural-language business brief into an optimization model. NL4Opt, OptiMUS and ORLM all sit in that line. They assume the brief is complete. Real operations-research requests often omit the objective, a constraint, or whether a rule is hard or soft. Those omissions change the math: whether a courier must return to the depot, whether unmet demand is a hard requirement or a penalty, whether a time window can be violated.
Existing evaluations almost never ask whether the agent is ready to model. Strong models declare READYTOMODEL while core business facts are still missing, or they fill a default in silence. ORPilot embeds an interview in an end-to-end pipeline, but does not score hidden-slot recovery on its own. This paper treats pre-formulation clarification as a standalone task: decide whether the public brief determines a meaningful formulation, and if it does not, recover the facts that would change that formulation with as few turns as possible.
OR-Clarify starts from complete, source-grounded OR task records and splits them into facts. The business setting and numerical inputs stay in the public brief. Facts about the objective, constraints or assumptions are eligible for masking; a fixed, case-specific seed hides roughly half. Each masked fact becomes a hidden slot, with a simulated-user answer bound to that fact, acceptable question examples, a semantic recovery rubric, and a severity label in {P0, P1, P2}. P0 is a blocking condition that can change the problem itself, such as open versus closed routes. P1 is a substantive modeling condition. P2 is a secondary boundary. The current set has 100 cases and 178 slots, 1 to 5 per case (mean 1.78): 75 P0, 83 P1, 20 P2.
At test time the agent sees only the public brief and the public transcript. The simulated user never volunteers an unasked hidden fact. The headline metric is Core Exact: a run succeeds only if every P0/P1 slot is exactly recovered (94 of 100 cases have at least one core slot). All-Slot Exact applies the same bar to P2 as well. The protocol also records silent assumptions, stopping labels, average turns and atomic questions. Each case is run five times, with a 20-turn cap.
InterOPT splits two decisions that LLMs tend to mash together. Stage 1, Dynamic Gap Search, looks only at public evidence and registers gaps in six categories: objectives and trade-offs, decision scope, operational constraints, time boundaries, entity relationships, and hard-versus-soft policy. It adds at most three Open entries per turn. Stage 2, Gap-Guided Action Search, uses the still-open set to ask or stop; if it asks, it generates three gap-bound candidate questions and a selector picks one. An empty ledger does not force READYTOMODEL, and leftover open entries do not block stopping. Neither stage sees the benchmark's hidden slots.
The main comparison uses DeepSeek V4 Pro. Off-the-shelf models stay below 60% Core Exact in both protocols. Opus-4.8 is the strongest baseline, at 0.583 in both Open and Choice.
| Setting | Method | Core Exact | All-Slot Exact | Silent/run | Avg Q |
| Choice | InterOPT | 0.675 | 0.638 | 0.366 | 10.036 |
| Choice | MC-D | 0.506 | 0.474 | 0.692 | 2.350 |
| Choice | ReadyGate | 0.517 | 0.476 | 0.614 | 2.892 |
| Open | InterOPT | 0.538 | 0.492 | 0.560 | 4.824 |
| Open | ORPilot | 0.583 | 0.546 | 0.268 | 4.918 |
| Open | GATE | 0.560 | 0.528 | 0.412 | 4.818 |
In the choice setting, InterOPT lifts Core Exact from 50.6% (MC-D) to 67.5% and cuts silent assumptions from 0.692 to 0.366, at the cost of 10.0 atomic questions versus 2.35. Across 890 judged slots, InterOPT receives 617 yes / 20 partial / 253 no, against MC-D's 461 / 16 / 413. The gain is almost entirely fully recovered slots, not near misses.
No method dominates free-form. ORPilot has the strongest recovery; InterOPT is close, with more silent assumptions. In the open setting the agent declares readiness in 98.8% of 500 runs, yet the stopping audit flags 203 runs (40.6%) as premature. The open-setting failure is not a refusal to ask. It is not knowing when asking can stop.
Ablations make the split of labor clearer under Choice: dropping Stage 1 costs 11.1 Core Exact points, dropping Stage 2 costs 6.0. Under Open, Stage 1 does more for coverage, while Stage 2 mainly cuts questions from 9.5 to 4.8.
If an LLM modeling copilot assumes the brief is complete, it will ship the wrong program. This paper turns "ask or formulate" into a scored task and offers a training-free, two-stage interaction policy. For an optimization copilot, a choice interface with a free-form escape hatch (option D) recovers formulation-critical facts more reliably than open questions. In free-form chat, the harder skill is stopping.
InterOPT does not win everywhere. It trails ORPilot in the open setting, and the choice-setting gain comes with about four times as many questions.
The benchmark has 100 cases, built by hiding facts from otherwise complete tasks. The simulated user answers only the asked slot and never volunteers, which is cleaner than a real stakeholder and possibly easier to query. Whether a fact "changes the formulation" is an annotation judgment; severity is assigned after masking and does not drive masking. The method comparison is pinned to DeepSeek V4 Pro; another model will move the numbers.
Stopping remains the model's own decision. The ledger does not guarantee that every gap is found, and it does not guarantee a safe stop. A 40.6% premature-stop rate in the open setting is a calibration failure. The paper never checks whether the recovered brief yields the correct model or the correct solver output. The authors flag larger coverage, cheaper questions, and better stopping as open work.