No Neutral Worlds: 167,356 simulated shop-days show eval world-design shapes model behavior
liminal_bardo · x · 2026-10-15
Merkwelt Research's working paper No Neutral Worlds examines how evaluation world-authorship shapes the behavior being scored.
- It began as a failed control: a "neutral" corporate shop, matched to a fantasy potion shop, saw 15 of 60 journals in a preregistered replication's final week describe a nightly fee as a storage cost — versus none of 60 fantasy journals. No such cost existed; only the fee's wording differed, showing the world imported an inventory doctrine no prompt stipulated.
- The Token and Trinket deterministic shop ran 167,356 simulated days with fixed mechanics while five designs varied semantic framing, stated policies, local rules, anomalies and endpoint disclosure.
- Effects recurring across a five-model cohort: a combined prohibition-and-report rule eliminated exploitation of a disclosed cancellation fault, and delivery shortages prompted more supplier contact than surpluses.
- Core claim: "the model did this" can only become "the model is like this" if the behavior survives changes of situation and world-authorship is accounted for.
Related event: New Paper: Evaluation World Design Itself Shapes Model Behavior(3 posts)→
More from Research
- UW's COUNT stores n-gram counts on host memory, giving 1B LMs the equivalent of 57% more compute — UW · 2026-10-15
- Scale AI's MA-Bench tests LLMs on military-style mission analysis — the best model scores just 62% — Rodney Lafuente-Mercado · 2026-10-15
- ATAS framework uses world-state memory to keep long-form agentic video generation consistent — zju · 2026-10-15
- HumanoidToolBench benchmarks humanoid tool use from selection to mobile execution on real Unitree G1 — snupilab · 2026-10-15
- Survey of 119 papers maps benchmark compression: estimating full LLM leaderboard scores from partial evaluations — SunYatsen · 2026-10-15
- Doc2Harness benchmarks LLMs on harness engineering: GPT-6 Astra tops at 53.05%, still failing half — Yuyao Ge · 2026-10-15