72 trials per model-harness pair on TB-Lite: no clear winner emerges
yb2698 · x · 2026-10-14
To capture meaningful signals, the author used the 24-task 'easy' subset of TB-Lite, repeating each task 3 times (72 trials per model×harness combination), measuring mean reward against mean input/output tokens and total agent runtime. No clear winner emerges from this comparison, prompting a deeper look at how prompt size evolves over turns.
More from coding & agent
- Startup launches cloud computers built for AI agents: 4 vCPUs in under a second — CurieuxExplorer · 2026-10-15
- User has Grok bot build its own email UI, and it just works — elonmusk · 2026-10-15
- YC F26's Papaya launches to auto-fix production AI agents via PRs — ycombinator · 2026-10-15
- Eval-driven development for RAG: keep eval cases in Git, never regress — Al_Grigor · 2026-10-15
- LlamaIndex launches OpenDocRouter with ParseBench benchmark comparisons — llama_index · 2026-10-15
- User reverse-engineers Grok's X news curation prompt and turns it into a reusable agent skill — dotey · 2026-10-15