Experiment details: Qwen3.5-9B and Qwen3.8-27B on Terminal-Bench Lite from open-thoughts
yb2698 · x · 2026-10-12
Details from the same thread: tests cover Qwen3.5-9B (24 easy tasks) and Qwen3.8-27B (48 easy+medium tasks) on Terminal-Bench Lite, which gives similar signal to Terminal-Bench 2 at much lower cost and works for smaller models. Links to the OpenThoughts-TBLite repo, a difficulty-calibrated benchmark for terminal agents.
More from coding & agent
- PageIndex SDK v0.4 claims tree indexing now cheaper than vector indexing by ~15% — solyarisoftware · 2026-10-13
- Economist: agentic coding lets you write your own Fortran optimizers, not just ask for Stata packages — Apoorva__Lal · 2026-10-13
- Tencent open-sources Agent Memory: local long-term memory, 61% fewer tokens, 48%→76% accuracy — solyarisoftware · 2026-10-13
- Testing AI agents with simulations: replay real traces, 50 curated sims beat 1000 random ones — HamelHusain · 2026-10-13
- How humans and Copilot agents split the work to build a GitHub Copilot beginner course — DanWahlin · 2026-10-13
- Engineers distill their AGI debate into an essay on self-verifying, continuously improving agents — dotey · 2026-10-13