Single ipython tool harness matches complex production agent frameworks for small models

Developer yb2698 published a series of experiments: benchmarking several open-source coding agent harnesses with Qwen3.5-9B on Terminal-Bench Lite, they found that a minimal harness using just one ipython session can match some complex production-grade harnesses. The core conclusion: "the less scaffolding, the better small agents perform." The experiment code is open-sourced on HuggingFace as ipython-harness.

Confirmed

• Qwen3.5-9B was chosen for the experiments: a 9B model with a suitable GGUF quantization needs only about 7GB of VRAM, so it can run locally on most consumer laptops—well suited for many repeated experiments
• The four open-source harnesses compared were opencode, Pi, Qwen-Code, and OpenHands, with wildly different built-in tool counts: Qwen-Code has 63, OpenHands 21, opencode 17, and Pi just 9
• The minimal harness, inspired by Alex Zhang's "Recursive Language Models" work, lets the LLM use a single ipython session directly; the author considers it a harness in its own right
• On the TB-Lite easy subset, 24 tasks × 3 repetitions each (72 trials per model×harness combination) measured average reward relative to input/output tokens and total runtime—no clear winner emerged
• Deeper experiments: focusing on 3 tasks with 20 repetitions each, re-run under three sampling settings (temperature=0 greedy decoding, model-recommended parameters, and harness defaults); OpenHands was the most stable across all sampling settings, while the minimal harness showed the most randomness, measured by the author across pass/fail flips, round-count consistency, and input token fluctuation
• The author re-ran single-task experiments with all ipython harness versions (v0→v3) to verify how harness improvements benefit small models' reward and token consumption (Qwen3.5-9B on the pandas-etl task)
• Environmental noise effects: irrelevant differences such as download-speed digits in pip progress bars (150.3 MB/s vs 129.3 MB/s) can cause small-model agent performance to wobble; the author fixed these artifacts with roughly 40 lines of regex. The analysis suggests small models are not robust to environmental artifacts in tool outputs, while large models are generally unaffected
• Prompt and tool-call analysis: Qwen-Code's per-round prompts are notably larger, rooted in its far bigger system prompt; Qwen-Code makes about twice as many read/search calls as other harnesses and leans more on built-in read tools; rounds per task are broadly even, with opencode slightly ahead; ipython shows no compression disadvantage on prompts in the first 30% of trajectories and pulls ahead later

Why it matters

• This series offers reproducible quantitative evidence on whether piling scaffolding onto small models helps, cautioning that details like environmental noise and system prompt bloat matter when building small language model harnesses; the open-sourced ipython-harness also lowers the barrier for others to reproduce and extend the experiments

2026-10-14 ~ 2026-10-14 · 16 related posts

• Harness experiment: cheaper model + guardrails beats frontier model at 33% less cost — blaizedsouza · 2026-10-14
• A single ipython tool harness ties complex production agent harnesses for small models — yb2698 · 2026-10-14
• [source] Single-tool harness matches complex production agents on Terminal-Bench Lite — yb2698 · 2026-10-14
• Experiments use Qwen3.5-9B: runs on ~7GB VRAM in consumer laptops — yb2698 · 2026-10-14
• Four open-source coding harnesses compared: tool counts range from 9 to 63 — yb2698 · 2026-10-14
• Inspired by Recursive Language Models: a bare ipython shell as the fifth harness — yb2698 · 2026-10-14
• [source] 72 trials per model-harness pair on TB-Lite: no clear winner emerges — yb2698 · 2026-10-14
• Qwen-Code's huge system prompt inflates every turn; no-compaction ipython grows largest late — yb2698 · 2026-10-14
• Qwen-Code reads files twice as often — the cause is its own system prompt — yb2698 · 2026-10-14
• Re-running the same agent task flips results: minimal harnesses are most stochastic — yb2698 · 2026-10-14
• 20-trial repeat experiments: OpenHands stays most stable across sampling settings — yb2698 · 2026-10-14
• For small models, ipython is all you need: single-tool harness ties production-grade scaffolds on TB-Lite — yb2698 · 2026-10-14
• Download-speed digits break small-model agents: a 40-line regex normalizer fixes it — yb2698 · 2026-10-14
• Small-model agent harnesses are fragile to environment noise — open-source ipython harness released — yb2698 · 2026-10-14
• Rerunning the single-task experiment across all ipython harness versions produced the benchmark plot — yb2698 · 2026-10-14
• [source] HuggingFace open-sources ipython-harness: a minimal ipython-only agent eval framework — yb2698 · 2026-10-14