tb-21 eval setup revealed: per-task 10-200 min limits, setup time excluded from agent clock
PawelHuryn · x · 2026-11-06
Pawel Huryn details the tb-21 evaluation setup: every task ran with its own default time limit (10 to 200 minutes), and the only change was a 3x increase to the setup time limit, since Claude Code and OpenCode npm installs ran past the default when several harnesses installed simultaneously. Setup time is not counted as agent time.
Related event: Terminal-Bench Tasks Fixed After Debian 11 Repo Failure(2 posts)→
More from coding & agent
- 28,849 trials and $4,746 later: a leaderboard ranks coding-agent harnesses on open models — HankYeomans · 2026-11-06
- vibe42ai ships a Linux client; users ask for security scanning built in — sujingshen · 2026-11-06
- SEO teardown: Amp appears in 0 of 28 AI answers, gets 1/7 the search traffic of peers — iannuttall · 2026-11-06
- Qwen3.8-27B as local coding agent burns 16k tokens thinking, never calls tools — previaegg · 2026-11-06
- Attendee's personal AI OS builds a live web artifact to manage conference notes and contacts — every · 2026-11-06
- Developer runs a pocket ConBot all day at a conference: badge-to-LinkedIn, auto tweet drafts — every · 2026-11-06