ByteDance's TraceDance auto-builds agent behavior benchmarks from real deployment traces
ByteDance · hf · 2026-09-29
TraceDance is an automated system that turns real agent deployment traces into targeted behavior benchmarks for developer-specified undesirable behaviors, going beyond fixed benchmark suites.
Key techniques: Anchor-and-Confirm combines programmable retrieval with candidate-level confirmation by a Flash LLM, while the Anchor Synthesis Loop generates and revises specifications for custom behaviors. Benchmarks use decision-point continuation to grade an LLM's next turn at a recorded decision point with a behavior-specific rubric—no reference answer or environment replay needed.
Experiments on coding and general tool use drew on 252,557 sessions, producing 107 benchmarks with 4,125 instances and fulfilling 95.3% of build requests. Human annotators confirmed 84% of sampled instances, and the automated grader's agreement with human pass/fail judgments matched inter-annotator agreement. Nine frontier LLMs averaged only a 26.7% pass rate, revealing persistent behavior weaknesses at decision points. The authors position TraceDance as a key component of a recursive self-improvement (RSI) loop.
More from coding & agent
- Generic API-catalog MCP server vs one tool per job: a context-bloat experiment — MandeepKaur9407 · 2026-11-16
- Berkeley paper: LLMs answer with stale values even when the update is in context — ombre-ai · 2026-11-16
- Effective HTML: 4.2k-star agent skills turn wireframes into self-contained interactive prototypes — tom_doerr · 2026-11-16
- skills-gate: one gate skill fronts 148 agent skills at just 389 tokens — Distinct-Pie2389 · 2026-11-16
- Open-weight Karpo-Decision-31B tops Decision Index at 73.9, one forward pass per answer — CodeByPoonam · 2026-11-16
- shadcn boosts @uisdk: one-shot UI rebuild that's instantly E2E-testable by AI — shadcn · 2026-11-16