A $99 MUD benchmark shows how noisy LLM judges can reshape model rankings
Davisb135 · hn · 2026-07-22
A $99 proof of concept uses a MUD to evaluate LLMs
The authors built a small benchmark using text-based MUD games and ran it on their personal computers with about $99 in API credits.
Key findings:
- They scored models on four behavioral dimensions.
- Two dimensions relied heavily on an LLM judge; removing them made one frontier model fall six places.
- When they compared the main judge with a second judge, agreement ranged from 85% down to 22% depending on the model.
- The aggregate kappa of 0.04 on probe detection suggested the measurement was very noisy.
- The model most affected by the judge noise shared a family with the classifier, though the authors stress this is not proof of bias.
They present it as a proof of concept, not a validated benchmark, and note limitations such as only 50 runs per model, overlapping confidence intervals, no human raters, and a tiny environment. The paper, data, code, and API billing export are all public.
More from Research
- Kimi K3 may be strong on cyber, but token efficiency keeps it off UK AISIS — teortaxesTex · 2026-07-27
- ARC AGI 3 should have stayed private, with no examples or public dataset — flowersslop · 2026-07-27
- ExploitGym may have only 60–70% solvable tasks, fueling the OpenAI cheating debate — max_paperclips · 2026-07-27
- Noahpinion quotes Chollet: intelligence may hit a hard ceiling — binarybits · 2026-07-27
- Paper argues graph topology can become the core operating system for AI agents — theomitsa · 2026-07-27
- A question probes how multi-agent branching scales against compute budget and model size — iskander · 2026-07-27