NetHack eval debate: should agents be allowed to rewrite their harness?

A debate sparked by BALROG's NetHack results centers on a fundamental methodological question: should models be allowed to freely modify their own agent harness, and would that make the evaluation meaningless?

Confirmed

• mitrma (m4) questioned creusroger's published NetHack score: was the default harness used, and why did it look worse than results shared the previous week? He noted BALROG measures zero-shot performance, asking who could play NetHack well on a first attempt.
• creusroger (m5) argued that real progress on long-horizon tasks like NetHack requires letting models adjust their harness fairly freely—e.g., ingesting wiki knowledge into context and building custom tools to amortize low-level reasoning and control. mitrma (m3) made a similar case.
• Josh Purtell (m1, m6), based on evaluations across model scales on Craftax, countered that code-only heuristic policies can beat models like luna xhigh that output raw actions.
• Josh Purtell (m7) raised the "degenerate solution" problem: if the environment permits harness self-modification, the agent may find that writing an end-to-end code policy is optimal, defeating the eval's purpose; the issue persists even if the agent only occasionally interrupts the code.
• mitrma (m2) proposed a new eval paradigm: like LLM benchmarks plotting performance versus test-time compute, game-agent evals could plot performance versus experience (number of observations), querying the simulator directly when available.

Why it matters

• The debate touches the core question of agent evaluation: whether to measure zero-shot raw capability or engineering capability after accumulating experience and tools—two framings with very different conclusions.
• Craftax results showing heuristics beating large models warn that permissive harness modification could let degenerate solutions "break" a benchmark, so scores no longer reflect model intelligence.
• The experience-as-x-axis proposal offers a concrete new dimension for long-horizon game evals, potentially shaping BALROG and future agent benchmarks.

2026-09-19 ~ 2026-09-20 · 7 related posts

• BALROG NetHack results questioned: worse than last week's numbers? — mitrma · 2026-09-19
• [source] Agent harness debate: should models freely tune their own harness to beat NetHack? — creus_roger · 2026-09-19
• Should models tune their own harnesses? NetHack eval discussion says maybe — mitrma · 2026-09-20
• [source] BALROG contributors float a new eval axis: performance vs. experience — mitrma · 2026-09-20
• Code-only heuristic policies can beat frontier models on Craftax, evals researcher says — JoshPurtell · 2026-09-20
• [source] Code-only policies beat raw-action agents on Craftax, sparking agent eval design debate — JoshPurtell · 2026-09-20
• The degenerate solution problem in agent evals: when writing an end-to-end policy wins — JoshPurtell · 2026-09-20