BALROG contributors float a new eval axis: performance vs. experience
mitrma · x · 2026-09-20
Contributors to the BALROG game-agent benchmark discuss a new evaluation dimension: just as LLM benchmarks plot test-time compute spend against performance, this paradigm could plot experience (number of observations seen) against performance. If you have the sim, you can query it and roll that into the test-time-compute measure—an axis they call currently unmeasured, hinting at BALROGv2.
The thread also proposes a bigger idea: to make progress on NetHack and games in general, models should be allowed to tune their harnesses quite freely—incorporating wiki knowledge and building custom tools that amortize low-level reasoning and control.
Related event: NetHack eval debate: should agents be allowed to rewrite their harness?(7 posts)→
More from Research
- Gradient-free weight editing cuts VLM FigStep attack success from 49% to 2% — lateinteraction · 2026-11-04
- Anthropic says it may imminently hit RSP threshold for AI-accelerated R&D, releases provable inference work — tobyordoxford · 2026-11-04
- Google scanned 1,033 real household products, packed into a 345 MB GLB for three.js — victormustar · 2026-11-04
- These 10 GPU optimization papers are why your model runs fast today — NandoDF · 2026-11-04
- INT8 isn't the problem: I benchmarked samplers on INT8 flow models — YakovSava · 2026-11-04
- Karpathy Boosts Open Hacker News Dataset: 10M+ Posts Usable as a Personal Knowledge Base — huangyun_122 · 2026-11-04