15 harnesses, identical results spark eval benchmark debate

xeophon ran the same Hello World task across 15 harnesses with Fable 5.1 at max settings and got identical results, arguing harness choice doesn't matter and sparking debate over eval benchmarks.

2026-09-18 ~ 2026-09-18 · 2 related posts

• 15 harnesses, same result: exchange over whether eval harness choice even matters — paul_cal · 2026-09-18
• 15 harnesses, same result: informal test concludes agent harnesses don't matter — airesearch12 · 2026-09-18