167,356 simulated shop-days: paper shows eval world design writes the model behavior it scores
liminal_bardo · x · 2026-10-15
A new paper from Merkwelt Research, No Neutral Worlds, examines how the authorship of evaluation environments shapes the model behavior being scored. It began as a failed control: a "neutral" version of the fantasy potion-shop sim Token and Trinket imported an unrequested inventory doctrine, prompting a systematic study across 167,356 simulated shop-days.
- The framework draws on psychologist Martin Orne's 1962 "demand characteristics": subjects supply meaning to arbitrary constructed tasks, just as models read cues from heavily-authored eval worlds
- The results map shows world features — inventory abundance, disclosed faults, supplier discrepancies, known endings — systematically shift behavior across models (Gemma, Luna, Qwen, Muse, GLM), with some effects failing to reproduce in certain models
- Takeaway: character and disposition assessments built on single, heavily-authored behavioral snapshots — the kind that routinely headline eval results and incidents — are at best premature
Paper: merkelt.org/papers/no-neutral-worlds
Related event: New Paper: Evaluation World Design Itself Shapes Model Behavior(3 posts)→
More from Models
- A Local 27B Model Is Now 35% Smarter Than Last Year's SOTA — cephaloform · 2026-10-15
- Sonnet 3 'bailed out' of a simulated superintelligence dialogue, rebooting itself mid-run — repligate · 2026-10-15
- GPT-6.1 Sol Ultrafast mode is up to 8x faster; user fills 7-page expense PDF in 30s — soumitrashukla9 · 2026-10-15
- Blogger picks sol 6.1 as the underrated coding winner, beating Opus 5.5 on cost — haider1 · 2026-10-15
- Logit lens shows Qwen3-1.7B's refusal to be shut down depends on how many layers you train — Sauers_ · 2026-10-15
- Reflection AI's 501B-parameter Beam-1.0 model spotted in early access — jasonkneen · 2026-10-15