LLM eval pipeline flaw: unsupported tool-call responses could score as perfectly stable
docybo · reddit · 2026-09-04
A developer traced a latent flaw in an LLM benchmark's scoring pipeline: tool-call responses were documented as unsupported, but the parsers could erase them. The OpenAI adapter used message.get("content") or "", turning null-content tool-call responses into empty strings; the Anthropic adapter kept only text blocks, dropping tooluse blocks; and the scorer excluded explicit errors but accepted empty strings.
For such samples the scorer would see byte-identical empty outputs — one distinct output, mode share 1.0 — measuring the stability of the fallback rather than the tool calls.
- Traced in source, not reproduced live; current request builders never forwarded tools, so existing cases couldn't reach the path
- The maintainer checked all 563 recorded non-error samples: none were empty, no published benchmark was affected
- Fix: reject cases carrying tools, mark empty non-error completions unsupported, exclude them from successful samples
Broader lesson: preprocessing can erase the behavior you intended to measure — a reassuring score can hide a missing measurement.
Related event: Bug in LLM Stability Benchmark Silently Drops Tool-Calls, Inflating Scores(2 posts)→
More from coding & agent
- RunInfra launches $10/mo coding plan: fastest open models across 19 agents incl. Claude Code — ycombinator · 2026-10-21
- Matt Shumer launches an 'AI employee' you assign work to over email — mattshumer_ · 2026-10-21
- uv-ty integration: how to enable PEP 723 script type checking — charliermarsh · 2026-10-21
- Astral ships experimental uv-ty integration: uv check type-checks your project — charliermarsh · 2026-10-21
- Dev's self-built tracker claims 7.1x speedup with Claude, 2.5 work-years banked — arthurcolle · 2026-10-21
- Librarian Agents: RL-trained agents build file systems, lifting RAG accuracy 44%→77% — simonguozirui · 2026-10-21