LLM eval pipeline flaw: unsupported tool-call responses could score as perfectly stable
docybo · reddit · 2026-09-04
A developer traced a latent flaw in an LLM benchmark's scoring pipeline: tool-call responses were documented as unsupported, but the parsers could erase them. The OpenAI adapter used message.get("content") or "", turning null-content tool-call responses into empty strings; the Anthropic adapter kept only text blocks, dropping tooluse blocks; and the scorer excluded explicit errors but accepted empty strings.
For such samples the scorer would see byte-identical empty outputs — one distinct output, mode share 1.0 — measuring the stability of the fallback rather than the tool calls.
- Traced in source, not reproduced live; current request builders never forwarded tools, so existing cases couldn't reach the path
- The maintainer checked all 563 recorded non-error samples: none were empty, no published benchmark was affected
- Fix: reject cases carrying tools, mark empty non-error completions unsupported, exclude them from successful samples
Broader lesson: preprocessing can erase the behavior you intended to measure — a reassuring score can hide a missing measurement.
Related event: Bug in LLM Stability Benchmark Silently Drops Tool-Calls, Inflating Scores(2 posts)→
More from coding & agent
- Claude Code self-hosted environments enter public beta — EricBuess · 2026-09-04
- Compared 6 AI Visibility Tools: $199 Ahrefs Actually Costs $974/Mo at Scale — Informal-Dust4499 · 2026-09-04
- AI agent River cleared 70% of vuln backlog in 11 days, merge rate 10% to 80% — jevon · 2026-09-04
- Databricks found $1.2M/year in wasted AI spend from 7 MCP-server bugs — matei_zaharia · 2026-09-04
- Open-sourced: training code for a mobile screenshot classifier (feeds, gallery, chats) — TheMoonMidas · 2026-09-04
- Model routing cuts LLM errors 46% at same cost, Martian study finds — SucceededMind · 2026-09-04