TerminalBench maintainer: eval data building is nuanced, join the open OSS effort
ajratner · x · 2026-10-21
- OfirPress (Princeton researcher) responded to an analysis criticizing TerminalBench: building this kind of eval data "is hard and filled with nuances," and the spirit of OSS projects like TerminalBench is to do it together in open, collaborative ways.
- He hopes the analysis team joins the channels the TBench community uses daily (repo, Discord, etc.) instead of working apart.
- Reflects the common tension between external benchmark critiques and upstream co-development.
More from Research
- Draft paper argues sparse MoE residual activations beat dense ones for interpretability — menhguin · 2026-10-21
- Discover AI: open-source guide with 51 interactive labs for learning how AI works — Zipidyzip · 2026-10-21
- Single neurons can steer LLM behavior — echoing a 2007 finding in real brains — HistedLab · 2026-10-21
- Dev rebuilds real Swiss terrain at 0.5m detail with 3,800 lines of code, no game engine — techartist_ · 2026-10-21
- UnsolvedMath dataset hits 17,614 open problems for LLMs, drawing mathematicians' ire — tak3sh8 · 2026-10-21
- mini-Lean-agent: bash-only coding agent solves 12/12 Putnam 2025 at $1.28 per problem — wellecks · 2026-10-21