How erased tool-call responses can fake a "perfect" LLM stability score
docybo · reddit · 2026-09-04
While auditing an LLM output-stability benchmark, the author found a latent gap between documented scope and the scoring pipeline: unsupported tool-call responses could be silently erased.
The chain:
- The OpenAI adapter's message.get("content") or "" turns null-content tool-call responses into empty strings
- The Anthropic adapter keeps only text blocks, dropping tooluse blocks
- The scorer excludes explicit errors but accepts empty strings
Result: byte-identical empty strings yield one distinct output and a mode share of 1.0 — measuring the fallback's stability, not the tool calls.
This was traced in source, not reproduced live; the maintainer checked all 563 recorded non-error samples, none empty, so no published results were affected. The fix rejects tool-carrying cases and flags empty non-error completions as unsupported.
Key lesson: preprocessing can erase the behavior you meant to measure — a reassuring score can hide a missing measurement.
Related event: Bug in LLM Stability Benchmark Silently Drops Tool-Calls, Inflating Scores(2 posts)→
More from coding & agent
- GPT-6 Astra team member admits launch issues: code slop and excessive confirmations — yanndubs · 2026-09-04
- Dev Builds Timed AI Mock Interviews in Your IDE, Shares What Worked With Claude Code — BeetleJuiceK9 · 2026-09-04
- What 1,137 agent writes taught this MCP server author about tool scoping and safety — QuanTradin · 2026-09-04
- diffusers-workflow: declarative JSON pipelines on Diffusers, agent-ready via MCP — dkackman11 · 2026-09-04
- Claude Code self-hosted environments enter public beta — EricBuess · 2026-09-04
- Compared 6 AI Visibility Tools: $199 Ahrefs Actually Costs $974/Mo at Scale — Informal-Dust4499 · 2026-09-04