A $99 MUD benchmark shows how noisy LLM judges can reshape model rankings

Davisb135 · hn · 2026-07-22

A $99 proof of concept uses a MUD to evaluate LLMs

The authors built a small benchmark using text-based MUD games and ran it on their personal computers with about $99 in API credits.

Key findings:

They scored models on four behavioral dimensions.
Two dimensions relied heavily on an LLM judge; removing them made one frontier model fall six places.
When they compared the main judge with a second judge, agreement ranged from 85% down to 22% depending on the model.
The aggregate kappa of 0.04 on probe detection suggested the measurement was very noisy.
The model most affected by the judge noise shared a family with the classifier, though the authors stress this is not proof of bias.

They present it as a proof of concept, not a validated benchmark, and note limitations such as only 50 runs per model, overlapping confidence intervals, no human raters, and a tiny environment. The paper, data, code, and API billing export are all public.

Original post →

A $99 MUD benchmark shows how noisy LLM judges can reshape model rankings

More from Research