Google Cloud's five-rule playbook for evaluating whether agent skills are worth their tokens
rseroter · x · 2026-11-05
Google Cloud's developer relations team published an engineering playbook for evaluating agent skills (SKILL.md), based on dozens of scenarios run against their own Google Cloud Developer Plugin with Gemini, Antigravity, and Claude Sonnet 5 in isolated Harbor pods on GKE Autopilot; they also contributed capabilities upstream to NVIDIA's open-source evaluator.
Rule 1: before rewriting your SKILL.md when evals show Δ=0, check trigger collisions and security — nine times out of ten the skill never loaded. Routers only inspect the YAML frontmatter name and description; past a dozen skills, under-triggering (describing implementation instead of intent) and sibling hijacking collide.
The other rules cover zero-contamination sandboxes (local workstations' cached credentials, shell history, and CPU contention mask baseline failures), measuring delta against real baselines, and keeping skills lean so they don't bloat the context window. Core point: pasting pages of CLI docs into markdown without an eval loop likely confuses a frontier model more than it helps.
More from coding & agent
- Kubernetes promotes AI Conformance to permanent SIG Architecture subproject — TerryTangYuan · 2026-11-05
- Code mode isn't about MCP: it's about whether tool results go to context or program variables — ritakozlov · 2026-11-05
- Agent Arena: frontier models show 55-minute gap on long-horizon agent tasks — arena · 2026-11-05
- Agent-based kernel optimization faces a fidelity-throughput tradeoff: random data misleads, cold starts cost 12.7 minutes — simran_s_arora · 2026-11-05
- Solo dev builds window-free OS where a local Qwen3.5 9B generates every screen — leonardo_c · 2026-11-05
- Apple team ships Siri Evaluations: don't skip evals when wiring App Intents into Siri AI — rudrank · 2026-11-05