Our model benchmark was mostly measuring a broken harness, not model capability
jonah_omninode · reddit · 2026-10-15
In an earlier OmniNode comparison, a small local model and a much pricier frontier model kept producing identical pass/fail results — seemingly a useful cost signal. On inspection the harness was broken: the check mostly looked for expected markers in output and couldn't tell whether either implementation actually worked, and in another run a reasoning model exhausted its response budget mid-answer, so a config issue masqueraded as model failure.
Neither result supported the conclusions drawn: the cheap model earned no general capability claim, and the expensive one never got a fair attempt. The author now separates invalid runs from incorrect answers before comparing scores, and keeps intentionally broken implementations that the checker must reject, to test the evaluator itself.
More from coding & agent
- Claude Code Pushes Urgent Security Update Across All Installations — umeshonai · 2026-10-23
- Claude cut a 2.6-hour video export to 20 minutes — but only after being told to rebuild the server — byhewar · 2026-10-23
- 3 flight-booking rules my AI agent checks that I'd never do by hand — Efistoffeles · 2026-10-23
- OpenJet claims Qwen3.8-27B coding runs 1.9x faster than llama.cpp on one RTX 3090 — JeremyCMorgan · 2026-10-23
- Building an autonomous SEO agent in Hermes with three skills: programmatic pages, Keywords Everywhere, DataForSEO — EXM7777 · 2026-10-23
- Building a sports betting AI agent with onchain execution and human approval by default: architecture and lessons — Dry-Comfortable-2678 · 2026-10-23