Ramp evals: native harnesses lift agent solve rates up to 10 points, but unevenly

xeophon · x · 2026-11-19

Ramp Labs finds the harness is a crucial lever on model behavior: native harnesses generally raise both solve rates and costs vs. a Bash-only toolkit, but unevenly. Codex 6.1 Terra and 6 Luna gain 10 points; newer 6.2 models see little change. Grok and DeepSeek gain a lot, Muse stays flat, and Fable 5.2 is actually cheaper in its native harness. Bottom line: benchmark agents with native harnesses or risk underestimating models.

Original post →

More from coding & agent

coding & agent channel →