Google's agent team: test discrete behaviors, not just benchmark scores
ombre-ai · reddit · 2026-11-02
A Google Developers Blog post on evaluating coding-agent harnesses argues that end-to-end benchmarks like Terminal-Bench and DeepSWE can move a composite score a few points without telling you why. Behavioral evals instead check discrete, observable actions—e.g., whether the agent asks a clarifying question on an underspecified prompt, or runs the local validator before declaring a build-file change done.
- The suite's main job is guarding against regressions: confirming a prompt tweak, tool schema change, or model upgrade didn't make the agent worse
- Suggested start: pick one recent failure, write a check for it, run batches since single runs are noisy
- OpenAI's Feb 2026 harness-engineering post attacks the same problem from the build side, feeding fixes back into docs, guardrails, and tooling
The author asks how practitioners handle tool-call assertions vs LLM-as-judge, and run-to-run noise.
More from coding & agent
- Qwen 27B orchestrates SAM 3 per tile: one H200 serves 12 patients in medical imaging — MaziyarPanahi · 2026-11-02
- LangChain to host live workshop on context engineering for agent memory — LangChain · 2026-11-02
- VibeCoding pain point: AI-built web tools ignore design systems and break on mobile — lxfater · 2026-11-02
- Open-source insect sticker library ships with agent-run generate-and-review pipeline — AlchainHust · 2026-11-02
- 19-year-old claims $750K profit from Claude Code-built trading bot started with $68 — aftahi_ai · 2026-11-02
- Solo dev vibe-codes a superhero MMO with Opus 5.5 in 6 weeks, burning 3B tokens — superpikapoo · 2026-11-02