Ramp evals: native harnesses lift agent solve rates up to 10 points, but unevenly
xeophon · x · 2026-11-19
Ramp Labs finds the harness is a crucial lever on model behavior: native harnesses generally raise both solve rates and costs vs. a Bash-only toolkit, but unevenly. Codex 6.1 Terra and 6 Luna gain 10 points; newer 6.2 models see little change. Grok and DeepSeek gain a lot, Muse stays flat, and Fable 5.2 is actually cheaper in its native harness. Bottom line: benchmark agents with native harnesses or risk underestimating models.
More from coding & agent
- Gradio and HF launch hackathon: reproduce NeurIPS 2026 papers with AI agents — Gradio · 2026-11-19
- Ramp SWE-Bench v2: top four coding models within points, but spend varies wildly — charliermarsh · 2026-11-19
- LangChain's deepagents hits 50M downloads, now the industry-standard agent harness — LangChain · 2026-11-19
- NVIDIA open-sources SkillEvaluator to vet AI agent skills before deployment — anthara_ai · 2026-11-19
- Resolve.ai publishes SRE-1 paper on AI agents for production incident investigations — iceman84 · 2026-11-19
- GPT-6.2 in Codex: split reviews hinge on your first line, says thread with 5 prompts — alexcovo_eth · 2026-11-19