MCP Evals Must Name the Client Config, Not Just the Server
Sussybaka800869 · reddit · 2026-10-02
A methodology discussion argues that the same MCP tool can work in one client and barely be discovered in another, and adding a skill that explains when to use it changes results again — so "the MCP server passed" omits much of the experiment.
Reef Infra's harness adapters provide pieces for testing these differences: an adapter renders the client's configuration, rules and skills, launches a headless task, and reads that client's session format. Pi, OpenCode, Claude Code, Codex, DeepSeek Harness and Hermes each have separate adapters.
Within one adapter, Reef Infra can compare an existing harness with a proposed change on the same tasks, e.g. testing whether a skill improves completion of a tool-backed task. Cross-client comparisons require separate experiments with matched models, tasks and tool access; the adapter does not make environments equivalent automatically.
The differences are consequential: Codex's Reef Infra setup rejects code extensions and disables hooks, plugins and web search during evaluation; Pi and OpenCode have extension/plugin surfaces; Claude Code can render a plugin entry but activation still needs the relevant marketplace or settings reference. MCP fixtures and success checks remain yours to provide, and the client configuration must be kept with the result because it determines what the measurement means.
More from coding & agent
- Computer use with Opus 5.5 shines on tedious web and form-heavy tasks — DimitrisPapail · 2026-10-02
- Reviewing AI code strains human attention—this dev is rebuilding the IDE around it — arpit_bhayani · 2026-10-02
- Manus agent reportedly runs 50 social media accounts at once with no human in the loop — anthara_ai · 2026-10-02
- LightMem-Ego upgrade brings real-time AI glasses memory with 0.42s latency — glenbeer · 2026-10-02
- Andrew Ng releases free 1-hour AI agent course with full roadmap — anthara_ai · 2026-10-02
- GitHub Copilot adds Dynamic Workflows: code-defined orchestration for multi-agent processes — PaulShellDev · 2026-10-02