Developer Uses Codex MCP to Audit MBPP and HumanEval, Finds Deep Issues
code_star · x · 2026-08-22
X user codestar shares his experience using Codex through an MCP server to audit benchmarks. He created 47 concurrent nac sessions and within 2 hours audited every sample of MBPP and HumanEval variants, correcting deep issues in prompts, test cases, formatting, and even upstream MBPP. He reminds: 'Remember, folks, always look at the data!'
Related event: Developer Uses Codex MCP to Audit HumanEval and MBPP Benchmarks(2 posts)→
More from coding & agent
- Grok Bot Ran a 12-Hour Data Analysis Nonstop, Handling API Rate Limits Itself — Daniel_Farinax · 2026-10-16
- Apple code signing drags for hours; official advice: just wait a week — pjausovec · 2026-10-16
- Hermes: a 4B model with 8K context got worse with more test-time compute, pointing to contextual reasoning — mangahomanga · 2026-10-16
- Flux 3 Klein's Default Templates Are Broken Without Its Prompt Upscaler — StrictMove5839 · 2026-10-16
- 21-person team ships hundreds of PRs weekly with AI writing 97% of the code — rseroter · 2026-10-16
- ClosedMCP: Set Up MCP Servers Once, Share Them Across Your Whole Team — Ok_Sprinkles1301 · 2026-10-16