Nebius analyzes 143K coding-agent trajectories: equal solve rates, very different behaviors
nebius · hf · 2026-11-16
Nebius releases "Interpreting Coding Agents Through Their Behavioral Trajectories," arguing that standard coding-agent benchmarks treat execution as a black box by scoring only binary success. The team analyzes 143,209 execution trajectories from 125 agent configurations spanning 117 models, and builds a rule-based parser mapping raw traces to high-level actions (repo search, file editing, test execution), audited against a six-model consensus reference on 500 events.
Key findings:
- Agents with similar solve rates can differ substantially in how they explore, modify, and verify code.
- Without using task outcomes, token counts, or tool names as features, aggregate behavioral profiles retrieve the corresponding agent stack across disjoint repositories and languages.
- Scaffold choice and reasoning effort systematically shift agents' search, editing, and testing behavior.
The deterministic parser, full annotation/analysis pipeline, and an annotated trajectory subset are released.
More from coding & agent
- Jay Alammar's An Illustrated Guide to AI Agents tops Amazon AI bestsellers with 300+ figures — JayAlammar · 2026-11-16
- Dev vibe-codes a retro CD crate app "My 20" in just a handful of prompts — evilrabbit_ · 2026-11-16
- Perplexity agents can cite every number in financial research to the exact filing line item — AravSrinivas · 2026-11-16
- VibePrevisBench: a benchmark turning scripts, photos or videos into executable Blender 4D scenes — _akhaliq · 2026-11-16
- Weaviate paper: tiered retrieval effort lets you trade accuracy, latency and cost per query — CShorten30 · 2026-11-16
- Reranking study: Zerank-2 + LLM second pass hits 0.47 nDCG@10 at ~$0.037/query — CShorten30 · 2026-11-16