Code-only policies beat raw-action agents on Craftax, sparking agent eval design debate
JoshPurtell · x · 2026-09-20
Continuing the NetHack harness debate, JoshPurtell shares eval experience on Craftac across model sizes: code-only heuristic policies can outcompete agents like luna xhigh emitting raw actions.
He warns that if the optimal solution is an agent writing an end-to-end code policy and running it against the env — or interrupting code exec only for a few scenarios — free harness modification may lead to degenerate solutions, questioning the eval design itself.
Related event: NetHack eval debate: should agents be allowed to rewrite their harness?(7 posts)→
More from coding & agent
- Orgo, a YC-backed platform giving AI agents their own cloud desktops, claims 1,000+ teams — Scobleizer · 2026-11-04
- Founder: Vibe-coded a PMF-successful product, then hired pro devs to fix the mess — sujingshen · 2026-11-04
- One prompt to audit cross-region traffic: dev cut trans-Atlantic latency by moving a service — DanielLockyer · 2026-11-04
- Agent silently completed only a quarter of a 200-module build and stalled waiting for confirmation — Promptmethus · 2026-11-04
- ChatGPT Agent mode can scan LinkedIn and Indeed and build a tailored job-application spreadsheet — nikola_mr64990 · 2026-11-04
- Blogger auto-applied to 500 jobs via ChatGPT and got 12 interview calls in 24 hours — nikola_mr64990 · 2026-11-04