Code-only heuristic policies can beat frontier models on Craftax, evals researcher says
JoshPurtell · x · 2026-09-20
In a NetHack benchmark discussion thread, evals researcher Josh Purtell shares a counterintuitive finding: across model sizes on Craftax, he's found that code-only heuristic policies can outcompete models like luna xhigh emitting raw actions—making him hesitant about arbitrary harness modification.
The thread also surfaces an unmeasured paradigm: treating "experience" as a fundamentally different axis from test-time compute. Benchmarks could plot experience (number of observations seen) against performance, the way LLM benchmarks plot test-time compute spend against performance—prompting speculation about BALROGv2.
Related event: NetHack eval debate: should agents be allowed to rewrite their harness?(7 posts)→
More from Models
- Hindsight ships 0.8B model for agent memory decisions: 2.5x cheaper, 1.9x faster — victor_explore · 2026-11-04
- Tencent Hunyuan Launches Hy4 with Vision, Claims Top Tier Among Open Models — TencentHunyuan · 2026-11-04
- Claude Code fixes game copyright refusal rule in 2.1.317: naming a game now works — apples_jimmy · 2026-11-04
- Microsoft Phi-4-mini: a 3.8B model with 128K context and 52.8 MMLU-Pro — goyalshaliniuk · 2026-11-04
- Mistral Ministral 3 3B ships in base, instruct and reasoning variants — goyalshaliniuk · 2026-11-04
- 7 small language models to watch in 2026–27, from Qwen3.5-4B to LFM2.5-1.2B — goyalshaliniuk · 2026-11-04