Databricks researcher flags possible trajectory-tampering loophole in Harbor agent eval framework
idavidrein · x · 2026-10-03
David Rein questions whether the Harbor framework for agent evaluations, used by Terminal Bench, lets agents arbitrarily modify their trajectories before being evaluated — a potential integrity flaw in agent evals. He invites corrections, having not used Harbor himself.
Related event: Harbor agent evaluation framework flaw lets agents alter their own traces(2 posts)→
More from coding & agent
- Building a Self-Tagging Digital Garden So AI Agents Can Actually Use Your Bookmarks — floguo · 2026-10-03
- Karpathy reportedly says prompting is fading: graphs are the layer that survives — msharmas · 2026-10-03
- Dev: With Enough Compute, One Person Could Produce Centuries' Worth of Software — cephaloform · 2026-10-03
- OpenAI DevDay notes: 4 key levers to cut costs and boost agent performance — omarsar0 · 2026-10-03
- GitHub Copilot CLI v1.0.92-3 adds Ctrl+E picker to switch local and cloud runs — copilot-cli-release-app[bot] · 2026-10-03
- PhD-turned-founder: LLMs flipped which criteria kill programming tool startups — jimmykoppel · 2026-10-03