ScrambleToolBench finds agents still brute-force tools after the map changes

declare-lab · hf · 2026-08-04

ScrambleToolBench shows agents fall back to brute force when tools drift

The paper introduces ScrambleToolBench, an interactive terminal benchmark designed to test whether agents can infer unfamiliar tool behavior from interaction alone.

What the benchmark changes

Tool schemas no longer reveal semantic cues.
Tasks follow a continuous curriculum.
The environment adds mapping drift, stochastic action failures, and temporal execution windows.

What the evaluation found

State-of-the-art language models can often make an initial discovery, but that does not carry over to robust adaptation.
When the tool mapping changes, agents fail to use deductive recovery strategies such as cycle tracing.
Instead, they show belief inertia or revert to exhaustive search.
More test-time reasoning often just increases expensive brute-force search rather than enabling better deduction.
Persistent memory reduces compounding errors, but still does not make agents efficient at inferring structural changes.

Takeaway

The benchmark exposes a gap between first-step tool discovery and real adaptation under changing environments.

Original post →

ScrambleToolBench finds agents still brute-force tools after the map changes

More from coding & agent