TraceTrial-Bench separates agent success from harness reliability: 31.1% of runs mislead
_akhaliq · x · 2026-10-28
TraceTrial-Bench scores agent performance and harness reliability separately across 95 tasks, 7 dimensions, 4 harnesses × 9 models and 3,420 runs.
Key finding: 31.1% of runs pass agent criteria while failing at least one harness criterion — task success doesn't mean the harness worked. The bench uses native traces, workspace artifacts and environment state for evidence-based evaluation, with failure attribution requiring evidence review before propagation is established.
More from coding & agent
- GitHub's Spec Kit pushes spec-driven development: 5 commands to plan before coding agents build — Arindam_1729 · 2026-10-28
- How LangChain built its design agent: taste comes from context, not the model — LangChain · 2026-10-28
- Dev predicts in 5 years we'll laugh at today's juggling of terminal and agent harness tabs — DavidWells · 2026-10-28
- Team reverse-engineers Claude Code's memory: simple design, with caveats — letandrewcook · 2026-10-28
- Multimodal GraphRAG models missing and contradictory evidence for storm damage assessment — JeremyCMorgan · 2026-10-28
- Vibe coding in real life: 20 hours of personal-project work done in 2 — gleech · 2026-10-28