Single-tool ipython harness matches complex frameworks in small-model agent experiments

Author @yb2698 ran a series of systematic experiments around "how much scaffolding small-model agents actually need": on Terminal-Bench Lite, a minimal harness using only a single ipython session can match some complex production-grade harnesses. The core conclusion: "the less scaffolding, the better the small agent." This has direct practical value for building agent frameworks around small language models, and the experimental harness is open-sourced on GitHub as huggingface/ipython-harness.

Confirmed

• The experimental model is Qwen3.5-9B; with a suitable GGUF quantization it runs locally in about 7GB of VRAM, well suited to many repeated experiments; the benchmark is Terminal-Bench Lite.
• The four open-source harnesses compared are opencode, Pi, Qwen-Code, and OpenHands, with wildly different built-in tool counts: Qwen-Code 63, OpenHands 21, opencode 17, and Pi just 9.
• The minimal harness, inspired by Alex Zhang's Recursive Language Models work, lets the LLM use a single ipython session directly.
• Using the TB-Lite easy subset (24 tasks × 3 repeats, 72 trials per combination), the author measured average reward versus input/output tokens and total runtime; the results make it hard to declare a clear winner.
• Repeated experiments (3 tasks, 20 runs each, under three sampling settings: temperature=0, model-recommended parameters, and harness defaults) show OpenHands is the most stable across sampling settings, while the minimal harness is the most random.
• Single-task experiments rerun across all ipython harness versions (v0→v3) verify the benefits of harness improvements for small models, comparing reward and token consumption.
• Behavior analysis: model behavior is broadly consistent across harnesses, mostly invoking the shell, but Qwen-Code makes about twice as many read/search calls as the others, stemming from its system prompt pushing heavier use of built-in read-type tools; Qwen-Code's per-turn prompts are noticeably larger, while ipython wins on prompts over the first 30% of trajectories (and is overtaken later).

Environment noise findings

• The author found that environment tool output varies due to irrelevant noise: when pip downloads the same package, the progress bar shows different download speeds (150.3 MB/s vs 129.3 MB/s) each time. Small models are not robust to such environmental artifacts, while large models are generally unaffected; cleaning the output with about 40 lines of regex fixes the resulting jitter.

Why it matters

• The experiments suggest that when building harnesses for small language models, one must carefully handle environmental artifacts in tool output and sampling randomness, and that "less scaffolding" may be more effective than stacking tools—offering a reproducible experimental path and open-source tooling for low-cost local agents (runnable on a consumer laptop).

2026-10-14 ~ 2026-10-14 · 15 related posts

• A single ipython tool harness ties complex production agent harnesses for small models — yb2698 · 2026-10-14
• [source] Single-tool harness matches complex production agents on Terminal-Bench Lite — yb2698 · 2026-10-14
• Experiments use Qwen3.5-9B: runs on ~7GB VRAM in consumer laptops — yb2698 · 2026-10-14
• Four open-source coding harnesses compared: tool counts range from 9 to 63 — yb2698 · 2026-10-14
• Inspired by Recursive Language Models: a bare ipython shell as the fifth harness — yb2698 · 2026-10-14
• 72 trials per model-harness pair on TB-Lite: no clear winner emerges — yb2698 · 2026-10-14
• Qwen-Code's huge system prompt inflates every turn; no-compaction ipython grows largest late — yb2698 · 2026-10-14
• Qwen-Code reads files twice as often — the cause is its own system prompt — yb2698 · 2026-10-14
• Re-running the same agent task flips results: minimal harnesses are most stochastic — yb2698 · 2026-10-14
• 20-trial repeat experiments: OpenHands stays most stable across sampling settings — yb2698 · 2026-10-14
• For small models, ipython is all you need: single-tool harness ties production-grade scaffolds on TB-Lite — yb2698 · 2026-10-14
• Download-speed digits break small-model agents: a 40-line regex normalizer fixes it — yb2698 · 2026-10-14
• [source] Small-model agent harnesses are fragile to environment noise — open-source ipython harness released — yb2698 · 2026-10-14
• Rerunning the single-task experiment across all ipython harness versions produced the benchmark plot — yb2698 · 2026-10-14
• [source] HuggingFace open-sources ipython-harness: a minimal ipython-only agent eval framework — yb2698 · 2026-10-14