Should models tune their own harnesses? NetHack eval discussion says maybe
mitrma · x · 2026-09-20
Discussing why models struggle at NetHack, an evals contributor argues that to make progress (in NetHack and in general) we should let models tune their harnesses quite freely—incorporating wiki knowledge and building custom tools that amortize low-level reasoning and control.
A BALROG contributor notes that BALROG measures zero-shot performance, quipping: who has ever beaten NetHack on their first attempt, even with wiki usage?
Related event: NetHack eval debate: should agents be allowed to rewrite their harness?(7 posts)→
More from Research
- Gradient-free weight editing cuts VLM FigStep attack success from 49% to 2% — lateinteraction · 2026-11-04
- Anthropic says it may imminently hit RSP threshold for AI-accelerated R&D, releases provable inference work — tobyordoxford · 2026-11-04
- Google scanned 1,033 real household products, packed into a 345 MB GLB for three.js — victormustar · 2026-11-04
- These 10 GPU optimization papers are why your model runs fast today — NandoDF · 2026-11-04
- INT8 isn't the problem: I benchmarked samplers on INT8 flow models — YakovSava · 2026-11-04
- Karpathy Boosts Open Hacker News Dataset: 10M+ Posts Usable as a Personal Knowledge Base — huangyun_122 · 2026-11-04