One default setting cost DeepSeek 16-21 points: the hidden trap in agent evals

TheZachMueller · x · 2026-11-13

Joel Niklaus, while comparing ten coding agents, found that when a model's config sets no output limit, OpenClaw defaults to an 8,192-token reply cap — costing DeepSeek-V4.1-Flash 16-21 points on the DeepSWE benchmark.

Mechanics:

His Harness Leaderboard v0.1.6 now passes each model's own limits through. His advice for agent evaluators: log why each reply ended — these runs threw no errors and are easily mistaken for genuine capability failures.

Original post →

More from coding & agent

coding & agent channel →