Rerunning the single-task experiment across all ipython harness versions produced the benchmark plot
yb2698 · x · 2026-10-14
The author notes the plot came from rerunning the single-task experiment across all ipython harness versions, showing Qwen3.5-9B on pandas-etl improving from reward 0.70 to 0.95 while input tokens dropped from 530k to 52k.
More from coding & agent
- Startup launches cloud computers built for AI agents: 4 vCPUs in under a second — CurieuxExplorer · 2026-10-15
- User has Grok bot build its own email UI, and it just works — elonmusk · 2026-10-15
- YC F26's Papaya launches to auto-fix production AI agents via PRs — ycombinator · 2026-10-15
- Eval-driven development for RAG: keep eval cases in Git, never regress — Al_Grigor · 2026-10-15
- LlamaIndex launches OpenDocRouter with ParseBench benchmark comparisons — llama_index · 2026-10-15
- User reverse-engineers Grok's X news curation prompt and turns it into a reusable agent skill — dotey · 2026-10-15