How erased tool-call responses can fake a "perfect" LLM stability score
docybo · reddit · 2026-09-04
While auditing an LLM output-stability benchmark, the author found a latent gap between documented scope and the scoring pipeline: unsupported tool-call responses could be silently erased.
The chain:
- The OpenAI adapter's message.get("content") or "" turns null-content tool-call responses into empty strings
- The Anthropic adapter keeps only text blocks, dropping tooluse blocks
- The scorer excludes explicit errors but accepts empty strings
Result: byte-identical empty strings yield one distinct output and a mode share of 1.0 — measuring the fallback's stability, not the tool calls.
This was traced in source, not reproduced live; the maintainer checked all 563 recorded non-error samples, none empty, so no published results were affected. The fix rejects tool-carrying cases and flags empty non-error completions as unsupported.
Key lesson: preprocessing can erase the behavior you meant to measure — a reassuring score can hide a missing measurement.
Related event: Bug in LLM Stability Benchmark Silently Drops Tool-Calls, Inflating Scores(2 posts)→
More from coding & agent
- Agentic SDLC: engineers shift from implementers to agent orchestrators as cycles shrink from weeks to hours — Pavan_Belagatti · 2026-10-21
- OpenAI's dots can now use company tools via Merge for Workforce with employee-bound permissions — shensi · 2026-10-21
- The topology of an LLM agent: where the model, loop, and tools run shapes cost and security — davidcrawshaw · 2026-10-21
- Agents with full-disk and accessibility access are an enterprise security nightmare, argues Runlayer — leveragedupside · 2026-10-21
- Acellera runs an unsupervised virtual lab of AI scientists on a real IL-23 molecular discovery program — gdefabritiis · 2026-10-21
- Pydantic author experiments with JS/TS and React support in the Monty sandbox — samuelcolvin · 2026-10-21