Code-only heuristic policies can beat frontier models on Craftax, evals researcher says
JoshPurtell · x · 2026-09-20
In a NetHack benchmark discussion thread, evals researcher Josh Purtell shares a counterintuitive finding: across model sizes on Craftax, he's found that code-only heuristic policies can outcompete models like luna xhigh emitting raw actions—making him hesitant about arbitrary harness modification.
The thread also surfaces an unmeasured paradigm: treating "experience" as a fundamentally different axis from test-time compute. Benchmarks could plot experience (number of observations seen) against performance, the way LLM benchmarks plot test-time compute spend against performance—prompting speculation about BALROGv2.
Related event: NetHack eval debate: should agents be allowed to rewrite their harness?(7 posts)→
More from Models
- Tencent Hunyuan teases upcoming LMArena debut — TencentHunyuan · 2026-11-04
- Insider guesses Astra 6.2 launches tomorrow, jokes to use up today's quota — bdsqlsz · 2026-11-04
- Same Claude model served by Bedrock one day and Anthropic the next, dev finds — georgos_pappas · 2026-11-04
- Abliterated GLM-5.3 pulls off red-team exploit in 3:41 for $0.02, stock model refuses — bratton · 2026-11-04
- Tencent Hunyuan unveils Hy4, open weights coming soon — heatOverflower · 2026-11-04
- OpenAI's next model won't be cheap, quips likening it to an 800B active-parameter frontier — teortaxesTex · 2026-11-04