Silent failures in Python MCP servers: SDK benchmarks on 1.30 vs 2.2, plus an audit plugin that caught 56/56 defects
ShipItFrog · reddit · 2026-10-31
The author's first MCP server told Claude a user's cards were gone when they weren't: if cards.json failed to parse — even from a UTF-8 BOM that PowerShell 5.1 adds — the server returned an empty deck as a normal result. To map what the Python SDK actually sends back, they ran deliberately broken servers on 1.30.0 and 2.2.0 and recorded raw JSON-RPC traffic:
- Returning {"ok": false, "error": ...} as a dict: both versions set isError: false, so the model treats the error as data and reports success — the most common trap; raise a tool error instead
- CallToolResult(iserror=True) (2.x spelling): flag dropped on 1.x, works on 2.x
- Uncaught exception internals: sent to the model on 1.x, masked on 2.x
- 9 KB print() on stdio: on 1.x the response is lost and the client times out
The author packaged the checks into a Claude Code plugin (a skill plus a read-only subagent reporting file, line and quoted code). On label-stripped broken test servers it found all 56 planted defects across four runs with no false positives; on two real servers, 17 of 18 known defects with one false positive. Each run costs about $1; run twice and merge for anything critical. Python only for now; repo includes raw test logs.
More from coding & agent
- LambdaTest's TestMu AI Mapped: From Test Cloud to Agent Verification — Designer_Hair_417 · 2026-10-31
- Open-source ultra-elements: 39 tunable spell VFX in pure three.js, LLM-extensible — majidmanzarpour · 2026-10-31
- How Boardy made CI 20x faster: PR waits drop from 40 minutes to 2 — inductionheads · 2026-10-31
- Product Engineering vs engineer-centric AI coding: ship all the way to paying users — PawelHuryn · 2026-10-31
- One start-here skill navigates you through the 43-skill agent workflow — PawelHuryn · 2026-10-31
- hbench visualizes agent trajectories — and caught DeepSeek reading benchmark answers — Psychological_Cry920 · 2026-10-31