How do you evaluate agents whose skill library keeps evolving between test runs?
tahaan · reddit · 2026-11-15
A developer building a skills-driven agent (110k tokens of tool definitions) describes how a background evaluation loop that auto-drafts new skills breaks his test harness: every run now faces a different skill set, making model or prompt comparisons meaningless. He proposes freezing skill versions between runs and asks how others handle evals for learning systems. Notably, all frontier models struggle with reasoning over large toolsets, with turns taking 3-15+ minutes.
More from coding & agent
- Raindrop lets coding agents create accounts before humans claim them, with write-only keys — New-Tailor4071 · 2026-11-15
- Write the eval runner before the agent: evals-first architecture with Ports and mock adapters — hwchase17 · 2026-11-15
- Developer builds a due diligence AI agent that screens VAT, LEI and sanctions lists — andrejusb · 2026-11-15
- A practical recipe: turn your weekly reports into a Grok Bot routine that only tracks decision-driving metrics — VibeMarketer_ · 2026-11-15
- Gear-evolved harness beats hand-tuned agents on Kaggle Titanic: 79.9% vs 72.96% — Pale-Training-6944 · 2026-11-15
- ProgramBench leaderboard: agents rebuild programs from binaries, Opus 5.5 tops at 18.5% — TheReedemer69 · 2026-11-15