BALROG NetHack benchmark debate: harness self-modification and degenerate solutions

A debate over agent evaluation methodology broke out around BALROG's NetHack results, centered on two questions: should models be allowed to freely modify their own harness, and does doing so render the benchmark meaningless?

Confirmed

• In m4, a skeptic (mitrma) questioned creusroger's published NetHack scores: Was the default harness used? Why are the results worse than what he shared last week? mitrma also noted that BALROG evaluates zero-shot performance, asking rhetorically who performs well the first time they play NetHack.
• creusroger (m5) made the core argument: to achieve real progress on long-horizon tasks like NetHack, agents should be allowed considerable freedom to adjust their own harness—for example, absorbing wiki knowledge into context and building custom tools to amortize low-level reasoning and control. mitrma (m3) took a similar position.
• Based on his own evaluation experience across model scales on Craftax, Josh Purtell (m1, m6) countered: a heuristic policy written purely in code can beat models like luna xhigh that output raw actions.
• Josh Purtell (m7) further raised the "degenerate solution" problem in benchmark design: if the environment allows harness self-modification, the agent may find that directly writing an end-to-end code policy to run the environment is the optimal solution, defeating the benchmark's purpose; the issue persists even if the agent only interrupts code execution in a few scenarios.
• mitrma (m2) proposed a new evaluation paradigm: just as LLM benchmarks plot performance against test-time compute spent, game agent evaluations might instead use "x = experience (number of observations), y = performance," and when a simulator is available, it can be queried directly and folded into the experience axis.

Why it matters

• The debate touches a fundamental question of agent evaluation: what should benchmarks measure—zero-shot raw capability, or engineered capability after accumulating experience and tools? The answers and conclusions for the two may differ drastically.
• The Craftax finding that "code heuristics beat large models" is a reminder: if harness modification is allowed freely, benchmarks may be "gamed" by degenerate solutions, and scores will no longer reflect the model's own intelligence.
• The "experience on the x-axis" proposal offers an actionable new metric dimension for long-horizon game task evaluation, potentially shaping the design of BALROG and future agent benchmarks.

2026-09-19 ~ 2026-09-20 · 7 related posts

• BALROG NetHack results questioned: worse than last week's numbers? — mitrma · 2026-09-19
• [source] Agent harness debate: should models freely tune their own harness to beat NetHack? — creus_roger · 2026-09-19
• Should models tune their own harnesses? NetHack eval discussion says maybe — mitrma · 2026-09-20
• [source] BALROG contributors float a new eval axis: performance vs. experience — mitrma · 2026-09-20
• Code-only heuristic policies can beat frontier models on Craftax, evals researcher says — JoshPurtell · 2026-09-20
• [source] Code-only policies beat raw-action agents on Craftax, sparking agent eval design debate — JoshPurtell · 2026-09-20
• The degenerate solution problem in agent evals: when writing an end-to-end policy wins — JoshPurtell · 2026-09-20