Benchmarking 56 fresh dependency bugs: MCP routing lifts Claude Code accuracy 68% to 91%
Available_Side_7134 · reddit · 2026-10-23
The Cairn Commons team shipped a read-only MCP that lets coding agents query container-verified, freshly-published dependency bugs and workarounds, benchmarked on 56 bugs across 27 packages.
- Problem: for bugs newer than training data and search indexes, Claude Code got local cause right 95% but only 68% on "is this known/fixed upstream," often falsely claiming no issue existed.
- Approach: zero-config Streamable HTTP MCP (no account), untrusted-evidence handling with masking, resilient error-string search with version fallback (e.g. Vercel ai 7.0.148 vs 7.0.150, fastapi 0.159.1 vs 0.161.0).
- Key routing lesson: attaching the MCP alone changed nothing; agents only used it 56/56 after MCP server instructions and CLAUDE.md said to call cairnsearch first on dependency failures.
- Results: known/fixed accuracy 68%→91% (p=0.004), fully correct 66%→89%, time -38%, web searches -71%, GitHub fetches -86%.
More from coding & agent
- Devs on Reddit swap tactics for stopping prompt regressions in AI agents — SleepFlimsy6963 · 2026-10-23
- Claude Code + ComfyUI MCP workflow for client work: paid models or open weights? — newaccount47 · 2026-10-23
- Social scientist: CS treats agentic development as a magic black box — RexDouglass · 2026-10-23
- Someone vibe-coded a Tesla Roadster game — before GTA VI — Jasonio · 2026-10-23
- Codex 0.164.0 'account/read failed' TUI error — fix is restarting the app-server daemon — bytebot · 2026-10-23
- Same native UI, two stacks: Swift build is ~80MB Mac-only, Zig build is ~10MB cross-platform — jedisct1 · 2026-10-23