Same agent harness, opposite rankings: TB4 benchmark drama exposes eval chaos
lunwang1996 · x · 2026-10-30
paradite flags that the TB4 story keeps getting wilder: vals ai and AA both used the mini-swe-agent harness, yet produced opposite rankings — vals ai: Opus 5.5 > Sonnet 5.5 > Kimi K3.1 Pro > Gemini 4 Carbon; AA: Kimi K3.1 Pro > Gemini 4 Carbon > Sonnet 5.5 > Opus 5.5.
lunwang1996 argues GDM likely isn't intentionally gaming benchmarks — agents are so complex that without spending a day digging through individual samples, you can't tell what you're actually evaluating. The takeaway: agent evals are highly harness-sensitive, making cross-leaderboard comparisons nearly meaningless.
More from coding & agent
- Vercel quietly launches "Agentic Infrastructure", positioning itself as the deploy target for coding agents — evilrabbit_ · 2026-10-30
- Garry Tan shows one-command GBrain: hosted Postgres+pgvector memory for any coding agent — garrytan · 2026-10-30
- Mollick on Opus 5.5: prompt less, build in checkpoints instead of one-shotting — eldonredwards · 2026-10-30
- Hugging Face Spaces' Agents button lets your AI agent use any demo for free — victormustar · 2026-10-30
- Atelico launches fully on-device game AI engine with LLMs, voice, and NPC behavior — nathanbenaich · 2026-10-30
- Hooking Kimi up to bloodwork via MCP: a Redditor mined years of lab results in one evening — Local-Parking5561 · 2026-10-30