Per-trace scoring misses contradictions across support handoffs in 1200-case eval
JitteryWorkman · reddit · 2026-10-24
In a 1200-case support agent eval, the author found a blind spot: 960 cases have a single trace and 240 span multiple. In 38 multi-trace cases, three linked traces each passed the existing policy scorer while making conflicting refund commitments — trajectory scoring evaluates each run independently, so prior commitments vanish from the judgment context, and a reviewer sees a different refund amount every turn.
The author is testing session-level scoring in Braintrust: dataset rows keyed by caseid with up to 64 traces available to a consistency scorer over merged context, preserving contradiction cases in a regression dataset and gating releases on the case-level result in CI. Open question: where the boundary should end when a support issue is reopened or handed off again.
More from coding & agent
- Factory tested Reflection's Beam pre-launch via Together AI for day-0 support — togethercompute · 2026-10-24
- Bench shows GPT-5.6/6 Luna crush multi-agent coordination while DeepSeek V4 fails — JoshPurtell · 2026-10-24
- Payman AI launches MCP server for natural-language payments and payee management — modelcontextprotocol · 2026-10-24
- Grok Bot adds secure vault so agents can log in and pay without seeing your secrets — XFreeze · 2026-10-24
- Comtrade MCP server pipes UN bilateral trade data into LLMs — modelcontextprotocol · 2026-10-24
- Brain Space Composer v0.2.2 adds layer editing and five new models including DeepSeek V4.1 Flash — pwlot · 2026-10-24