Stanford/MIT paper: same LLM shows up to 6x gap depending on the harness
rohanpaul_ai · x · 2026-09-13
A Stanford and MIT paper, "Meta-Harness: End-to-End Optimization of Model Harnesses", argues AI performance depends not just on the model but on the surrounding system code — the harness that decides what to store, retrieve, and show the model. With the same underlying LLM, changing the harness creates up to a 6x performance gap on the same benchmark.
The paper introduces Meta-Harness, an outer-loop system that automatically improves harness code by giving the optimizing agent rich access to prior code, logs, and execution traces via a filesystem-like setup, instead of just a score or summary.
Key results:
- Online text classification: +7.7 points over strong SOTA context management while using 4x fewer context tokens;
- Retrieval-augmented math reasoning: +4.7 points on average across five held-out models on 200 IMO-level problems;
- Agentic coding: discovered harnesses beat strong hand-engineered baselines on TerminalBench-2.
The takeaway: attention should shift from "which model is best" to how the whole AI system is designed, since harness design governs reliability, tool usage, context management, and failure recovery.
More from Research
- Mysterious AI system "Odin" credited with proving the Komlós conjecture, paper on arXiv — badumtsssst · 2026-09-13
- Schulman cites model trained only on pre-1930 text beating Claude 3 Opus after distillation — victor_explore · 2026-09-13
- Researcher predicts LLM community will rediscover fixed-point iteration in looped transformers — gklambauer · 2026-09-13
- UEC Graduate Course on Interactive Systems by Prof. Kajimoto Releases All Lecture Materials and Videos — plopesresearch · 2026-09-13
- Robotics academics in despair as Astra, Fable, Muse zero-shot benchmarks — LerrelPinto · 2026-09-13
- Intuitive RL blog series opens with Manim-animated take on contraction operators — AvivTamar1 · 2026-09-13