One default setting cost DeepSeek 16-21 points: the hidden trap in agent evals
TheZachMueller · x · 2026-11-13
Joel Niklaus, while comparing ten coding agents, found that when a model's config sets no output limit, OpenClaw defaults to an 8,192-token reply cap — costing DeepSeek-V4.1-Flash 16-21 points on the DeepSWE benchmark.
Mechanics:
- DeepInfra, the serving provider, cut thinking at 4,916 tokens (60% of 8,192), while the model kept thinking in the visible reply;
- When a reply hit 8,192 tokens, OpenClaw took the half-finished sentence as the final answer and stopped;
- This ended 57% of runs at low reasoning effort and 69% at high effort; only 3 of 419 runs passed;
- GLM-5.3-Flash, whose provider ignored the cap, gained nothing from the fix.
His Harness Leaderboard v0.1.6 now passes each model's own limits through. His advice for agent evaluators: log why each reply ended — these runs threw no errors and are easily mistaken for genuine capability failures.
More from coding & agent
- Is Haskell 'unsafe' too? A debate on types vs verification in the LLM era — satnam6502 · 2026-11-22
- Lessons from building a desktop AI agent that sees your screen and drives the mouse — SpeechAggravating453 · 2026-11-22
- MCP server checks if a pay-per-call x402 API is worth paying, from 24,766 real paid calls — biryaniAssasin · 2026-11-22
- Walmart and Muse to launch first agent-to-agent agentic commerce integration over MCP — AccBalanced · 2026-11-22
- scientific-toolkit: one MCP server fronting nine repos that refuses to fabricate results — Emergency-Tutor-7 · 2026-11-22
- Zero-degradation Strata tuning: fixing layer_split lifts decode ~10-15% and halves CPU expert misses — PathfinderTactician · 2026-11-22