Agent harness engineering takes center stage as same model shows 6x performance gap

Engineering communities have been intensely discussing how decisive agent harnesses (the systems wrapped around models) are for Agent reliability. The emerging consensus: a powerful model doesn't make a reliable Agent—what really determines performance is the engineering system built around the model.

Confirmed

• A University of Toronto study drawing on 57 papers found that the same model can vary up to 6x in performance depending on harness design. A Marmelab author combined these papers with hands-on experience to distill harness engineering best practices, and suggests mining correction patterns for agents from historical sessions.
• Shalini Goyal published a long-form piece systematically covering 5 harness engineering techniques: bounded control loops (a plan → act → observe → evaluate loop with iteration caps and stopping conditions to prevent infinite loops and wasted resources); engineered context and state management (task-relevant context only, maintaining working memory and task progress, persisting checkpoints so interrupted runs can resume); safe tool access (least privilege, input/output validation, human approval for sensitive operations); verification and evaluation (validating against task requirements, running tests and independent checks before completion, tracking real-world success rates); observability and recovery (logging process details such as tool calls).
• Goyal stresses that "the Agent says it's done" doesn't mean the task was actually done correctly—independent verification is required.
• Goyal also compiled a set of learning resources, including Microsoft Agent Framework / Agent Harness docs, Anthropic's agent evaluation materials, and related papers.

Why it matters

• The University of Toronto's 6x performance gap shows that, for a fixed model capability, harness engineering is one of the highest-leverage investments for improving Agent performance.
• This framework reframes "Agent reliability" from a model-selection problem into a systems engineering problem: the model provides intelligence while the harness provides structure, control, and reliability—with direct implications for how teams allocate technical effort and organize work.

2026-10-30 ~ 2026-11-01 · 9 related posts

• [source] Agent harness engineering: lessons from 57 papers — same model, 6x performance gap — Marmelab · 2026-10-30
• [source] 5 Harness Engineering Techniques That Turn LLMs Into Reliable AI Agents — goyalshaliniuk · 2026-11-01
• Build a Bounded Control Loop to Keep AI Agents From Spinning — goyalshaliniuk · 2026-11-01
• Bounded Loops and Context Engineering: First Steps to Reliable Agents — goyalshaliniuk · 2026-11-01
• Least Privilege and Checkpoints: Two Pillars of Reliable AI Agents — goyalshaliniuk · 2026-11-01
• Secure Tool Access and Result Verification for Dependable AI Agents — goyalshaliniuk · 2026-11-01
• An Agent Saying 'Task Completed' Isn't Proof It Was Done Right — goyalshaliniuk · 2026-11-01
• [source] The Model Provides Intelligence, the Harness Provides Reliability — goyalshaliniuk · 2026-11-01
• Agent Harness engineering resources: Microsoft Agent Framework, Anthropic evals, papers — goyalshaliniuk · 2026-11-01