BALROG NetHack results questioned: worse than last week's numbers?
mitrma · x · 2026-09-19
In the BALROG NetHack eval discussion, a participant questions creusroger's posted results: were they run with a default harness, and why do they seem worse than the results he was sharing last week? The point is made that BALROG assesses zero-shot performance—but who has ever beaten NetHack on their first attempt, even with wiki usage?
Related event: NetHack eval debate: should agents be allowed to rewrite their harness?(7 posts)→
More from Research
- Gradient-free weight editing cuts VLM FigStep attack success from 49% to 2% — lateinteraction · 2026-11-04
- Anthropic says it may imminently hit RSP threshold for AI-accelerated R&D, releases provable inference work — tobyordoxford · 2026-11-04
- Google scanned 1,033 real household products, packed into a 345 MB GLB for three.js — victormustar · 2026-11-04
- These 10 GPU optimization papers are why your model runs fast today — NandoDF · 2026-11-04
- INT8 isn't the problem: I benchmarked samplers on INT8 flow models — YakovSava · 2026-11-04
- Karpathy Boosts Open Hacker News Dataset: 10M+ Posts Usable as a Personal Knowledge Base — huangyun_122 · 2026-11-04