Turn count misleads agent benchmarks: DEPTH measures sequential dependency depth instead
Yu Jiang · hf · 2026-11-04
A paper from Notre Dame, IBM Research, CMU and others examines "depth" in long-horizon terminal task benchmarks.
- Finding: synthesis methods usually measure task length by turns a model needs and difficulty by LLM-set labels, but turn count largely reflects model habits rather than task demands, and labels barely predict pass rates
- DEPTH: measures and synthesizes tasks by sequential dependency depth instead, producing genuinely harder benchmarks
- Fine-tuning agents on these harder tasks improves performance
Methodological lessons for agent evaluation and training data synthesis.
More from coding & agent
- EvoTeam evolves multi-agent team organization and skills from execution experience — JLU-Deep · 2026-11-04
- Switch-SWE benchmark shows coding agents lose focus when switching unfinished tasks — SJTU · 2026-11-04
- Claude Code trick: one command audits and rewrites your prompts for Opus 5.5 — Arindam_1729 · 2026-11-04
- macOS 27.3 adds native Markdown rendering to Quick Look, Siri skills use SKILL.md — timigod · 2026-11-04
- Fan builds draggable "Tibo calibrator" meme page with Nano Banana and RIFE for $0.24 — GlobalProcedure1731 · 2026-11-04
- Valid tokens, allowed actions, zero intent checks: the agent security gap nobody solves — Cytersmyz · 2026-11-04