ai-rulez author shows how a single success score hides MCP skill-discovery failure modes
Goldziher · reddit · 2026-11-16
The maintainer of ai-rulez (MIT-licensed agent config tool) shares a reproducible Claude Code experiment measuring MCP tool discovery, skill loading, and answer quality separately.
- Setup: 8-skill fixture; conversations move between sibling directories with identically named worksheets but different workflows. Each positive turn independently records whether findskill got the right path, whether loadskill returned the right body, and whether the response contained expected code; negative controls ask unrelated questions or unavailable workflows.
- Ablations: after a pilot accidentally omitted generated root instructions, four arms (stub-only, released root instructions, tools loaded upfront, revised guidance) were compared. Final config: Claude completed all three steps on 12/12 positive turns with zero calls on 6 unrelated turns.
- Key blind spot: in 5 of 6 wrong-workflow controls, the model still loaded a neighboring skill after the server correctly filtered out the unavailable one — invisible to path-only metrics.
Report, 192 sanitized turns, and reproduction commands are public; the author asks what negative control would catch models that treat the top search hit as applicable by default.
More from coding & agent
- WebArena-Pro: a 300-task multimodal benchmark across 20 self-hosted apps for web agents — McGill-NLP · 2026-11-16
- Filmworkbench v0.1.2: one-prompt semi-automated short films with Claude and ComfyUI — Eastern-Ad9515 · 2026-11-16
- Stop prompting LLMs for JSON: PowerShell module Jev routes with decision models — dfinke · 2026-11-16
- I killed my 30-minute weekly sync with a decision log my agents read before every run — Humaux · 2026-11-16
- A video editor tested free MCP-connected AI editing tools: invideo, ChatCut, FableCut — lol_you_again · 2026-11-16
- Web design is turning into WebMCP: sites exposing registerTool() interfaces for agents — galratner · 2026-11-16