Nebius analyzes 143K coding-agent trajectories: equal solve rates, very different behaviors

nebius · hf · 2026-11-16

Nebius releases "Interpreting Coding Agents Through Their Behavioral Trajectories," arguing that standard coding-agent benchmarks treat execution as a black box by scoring only binary success. The team analyzes 143,209 execution trajectories from 125 agent configurations spanning 117 models, and builds a rule-based parser mapping raw traces to high-level actions (repo search, file editing, test execution), audited against a six-model consensus reference on 500 events.

Key findings:

The deterministic parser, full annotation/analysis pipeline, and an annotated trajectory subset are released.

Original post →

More from coding & agent

coding & agent channel →