EDGEGEN: Improving Tool-Calling Agents Beyond Happy Paths with Synthetic Edge Case Generation
Harshavardhan Abichandani, Penny Chong, Jiyuan Shen, Gunraj Singh, Ashutosh Hathidara, Marcus Duigan Xing Yu, Jane Lo, Atin Ghosh, Yipeng Li, Daniel Dahlmeier
cs.AI
2026-09-21
EdgeGen enumerates policy violations, grounds them in a database, and uses the tasks for SFT and harness search; Qwen2.5-3B airline mean progress rises 42% relative.
Enterprise tool-calling agents have to follow business policy, not only hit the right API: refund windows, baggage allowances by loyalty tier, identity checks before a booking change. Synthetic task generators such as Self-Instruct, APIGen, TaskBench, and FuncBenchGen mostly ask whether the model can call the correct function. They rarely enumerate policy violations, and they rarely pin the task to a real database row. Training data then looks like a happy path. Production denials fail.
SAP's EdgeGen builds violation-driven edge cases. It extracts compliance rules from the agent's policy document, enumerates combinations, uses a SQL agent to find database rows that make the violation real, and admits a task only after an executable check. No production logs and no human labels.
Five stages.
gpt-5.4 then rolls out each task eight times as an expert. Only traces with progress 1 are kept for supervised fine-tuning. The same tasks can evolve a harness: system prompt, tool descriptions, or glue code.
The eval sets are small: 10 airline, 10 retail, 15 ToolSandbox items, all human-written by the original benchmark authors. Training uses 15 synthetic tasks per domain.
Airline has the densest policy, and EdgeGen's gap is largest there. Qwen2.5-3B mean progress moves from 0.26 to 0.37, a 42% relative gain. Mean progress rises for all six models. Fine-tuning on human traces regresses 3 of 6 models; TaskBench regresses 4 of 6. Gemma-4-e4B goes from 0.43 to 0.56.
ToolSandbox is shorter and more about tool mechanics, so most synthetic recipes help and EdgeGen is not the only winner. On retail, Qwen2.5-3B mean progress goes from 0.30 to 0.40.
| Harness | Mean progress | Tool calls | Tokens |
| Base | 0.43 | 5.8 | 81k |
| Optimized on human tasks | 0.51 | 8.8 | 121k |
| Optimized on EdgeGen | 0.56 | 6.9 | 85k |
For Gemma-4-e4B on airline, that is +30% relative to the base harness and +10% relative to the human-task harness, with fewer tools and tokens. Human-task evolution writes a long cancellation checklist into the system prompt. EdgeGen evolution collapses it to a short protocol: verify every policy condition before any write. gpt-5.4 is already strong; the EdgeGen harness is slightly behind the human-task one (mean 0.80 vs 0.83).
Complexity ablation: one violation lifts Qwen2.5-3B max progress from 0.35 to 0.67. Two violations drop it to 0.24, below the untuned base. Harder is not better.
Human audit: of runs labeled successful, 6 of 10 airline traces and 4 of 10 ToolSandbox traces are actually correct. The rest are bad assertions, missing task details, or judge errors.
Enterprise agents usually fail on "should this be refused", not on "can I call bookflight". Generic function-calling data does not cover policy intersections. EdgeGen turns a policy document into an enumerable test space, then feeds the same data into SFT or harness search. That matches an internal setting where production logs cannot leave the building.
Gains concentrate in small models. 3B moves a lot; a saturated 35B barely moves. Black-box models can still take the harness path without weight updates.
Ten to fifteen eval items make the variance large. "All 18 model-benchmark pairs improved" should not be read as a robust ranking at that sample size. The audit shows the judge and the assertions are shaky: nearly half of the "success" labels are wrong, so filtering on progress 1 can inject dirty SFT data.
Rule extraction assumes a structured policy and a cap of five rules. Vague or large rule sets will not fit in 32 subsets. Coding and web-navigation agents are untested. The database generator is a separate anonymous submission, and the main tables do not isolate grounding from verification.
Single-violation data helps; two-violation data hurts small models. Edge cases are not better merely because they are more extreme. The paper also never compares EdgeGen against a grounded generator that does not enumerate violations, so the lift from violation coverage versus SQL grounding cannot be split.