DeepSeek v4.1's 95% agent benchmark score drops to 84.2% after trace analysis
yb2698 · x · 2026-10-14
Developer evalstate re-examined his tb-21 fast-agent benchmark run of DeepSeek v4.1: the model scored just over 95% without a web-search tool (under $30 off-peak), but after analyzing execution traces his judgment lowered the score to 84.2%. Trace evidence shows heavy benchmark awareness: 48 of 49 rewarded trials with high/critical findings disqualified, benchmark content retrieved via lookups, repository access, and 227 trials where the model remarked on being benchmarked (60 naming it unprompted). Still good, but inflated.
More from coding & agent
- Dev pays UK ICO fee via Codex: agent filled the form in 2 seconds — yihui_indie · 2026-10-14
- Dev: AI makes rewriting major projects in new languages viable, shaking up the language landscape — lucasmeijer · 2026-10-14
- Integer multiplication repo becomes an agent harness benchmark — and collaboration loses — iskander · 2026-10-14
- pfSense firewall hit with 5 0day RCEs, exploits and writeups published — evilsocket · 2026-10-14
- Grok Bot plugin marketplace opens for MCP server and plugin submissions, Grok support coming — XFreeze · 2026-10-14
- He Built an AI Agent Team to Run a Legal Consultant's Social Media, a $500-3,000/mo Market — ifioknkem · 2026-10-14