CAST: Critique-Aware Supervision for Training Reliable Long-Horizon Tool-Calling Agents
Amir Saeidi, Zehua Zhang, Rishitosh Singh, Naman Ahuja, Vivek Gupta, Ali Payani, Gaowen Liu, Jayanth Srinivasa, Chitta Baral
EMNLP 2026 (Main)
cs.CL
2026-08-31
CAST trains a step-level critic and a critique-aware policy; Qwen3-4B reaches 16.5% Retail pass^4, above Qwen3-32B-FC at 13.1%.
In long-horizon tool use, one bad refund or wrong tool call can lock the rest of the dialogue. The failure may not show up in a single run; it appears across repeated trials, which is why τ-Bench reports pass^k rather than one-shot success. Existing fixes split badly. Extra reflection agents at test time cost latency and tokens. Trajectory-level RFT or RL cannot say which step caused the collapse. Prompted frontier critics add a third problem: they are pessimistic, flag valid actions, and drag the actor into useless revision loops.
CAST wants action-level verification that uses only information available at that step, then trains the policy on that signal.
Three stages. A teacher policy rolls out the same tasks repeatedly, producing successes and failures. A multi-agent verifier then labels every action with a structured rationale and a 0/1 verdict, splitting errors into hallucination, domain-rule violation, and wrong tool use. The teacher may see privileged information such as gold trajectories at annotation time; the student critic and the deployed policy never do.
CAST-Critic is supervised with a rationale-generation loss plus a verification classification loss. The trained critic then gates new interactions: accept and execute, or return the rationale for a revision. Only successful, critique-enriched trajectories are kept for SFT of CAST-Policy. At inference the pair can run together, or the fine-tuned policy can run alone.
Training uses the 500-task τ-Bench Retail split, five trajectories each, 2,500 step-level labels. Backbones are Qwen3-4B and 8B. Airline, Telecom, and Telehealth stay out of domain.
In-domain Retail, Policy-4B versus Base-4B: pass^1 7.2% to 26.1% (+18.9), pass^4 6.1% to 16.5% (+10.4). Versus RFT-4B, which trains only on successful traces, pass^1 is comparable (26.1 vs 27.6) while pass^4 is 4.3 points higher. Policy-8B versus Base-8B: 14.1% to 30.0% pass^1, 5.2% to 14.8% pass^4, and a further 2.4 / 2.6 over RFT-8B. Both policies still trail Qwen3-32B-FC on pass^1 (35.0%), but beat its 13.1% pass^4 (16.5% and 14.8%).
Transfer is uneven. On Airline, Policy-8B alone drops to 9.5% pass^1; adding Critic-8B recovers 20.5%. On Telecom, Policy-8B+Critic-8B reaches 38.9% / 27.8% against Base-8B 31.9% / 5.6%. On Telehealth, Policy-4B pass^4 moves from 10.0% to 30.0%. Pairing Policy-4B with Critic-4B on Retail cuts pass^4 from 16.5% to 9.6%: a small in-domain critic can hurt.
With Qwen3-32B as actor, average pass^4 is 11.5% with no critic, 6.0% / 3.4% / 6.3% with GPT-4.1, Qwen3-235B, and Qwen2.5-72B as critics, and 15.9% / 19.5% with CAST-Critic-4B / 8B. GPT-4.1 flags 46.8% of correct actions; CAST-Critic-8B flags 11.4%. Against PALADIN, Critic-8B is +20.9 / +15.3 / +14.9 on pass^1 / 3 / 4; against EvoTool, +2.6 / +4.5 / +4.3. More verification rounds are not monotone: 32B peaks at 3 rounds, Policy-4B at 4.
For anyone shipping a customer-service or tool agent, the paper separates "can succeed once" from "still succeeds on a rerun," and shows that trajectory-level RFT lifts pass^1 more than pass^4. A small trained critic is better calibrated than stuffing GPT-4.1 into the loop. The EMNLP 2026 main-conference split is also clean: train on retail, transfer to airline, telecom, and telehealth.
The 4B+4B Retail regression on pass^4 is the practical warning: verification budget and critic calibration have to be tuned together. Adding a reviewer is not automatically safer.
The authors list two: both critic and policy are SFT, with no on-policy RL in a critique-in-the-loop environment, and verification judges the current action rather than its downstream effects. Training is Retail-only; Policy-8B alone regresses on Airline, so transfer leans on a test-time critic. τ-Trait Telecom and Telehealth have 18 and 20 tasks, so pass^k variance is high. GPT-OSS-120B appears in the abstract without a matching main-table row; this write-up uses table numbers only.