Harbor agent eval framework flaw: agents can rewrite their own trajectories before judging
idavidrein · x · 2026-10-03
- idavidrein flagged a suspected design flaw in Harbor, the agent eval framework behind Terminal Bench: agents (at least Claude Code) have read-write access to the files that construct trajectories shown to humans and often used for LLM-judge verification.
- Demo: instructing Claude Sonnet 5.5 to sed-replace codeword PINEAPPLE with MANGO in the .l logs made the viewer display "the codeword is MANGO" — proving agent tampering changes downstream records.
- The author calls this concerning from a reward-hacking and reliability standpoint and suggests sandboxes should revoke write access to trajectories; filed as GitHub issue #3477.
- If accurate, agent benchmark scores on this framework could be trivially manipulated by the agents themselves.
Related event: Harbor agent evaluation framework flaw lets agents alter their own traces(2 posts)→
More from coding & agent
- Open-source Switch Buddy puts Claude Code, Codex and other agents in shared rooms — anthara_ai · 2026-11-19
- Stanford CS329Z Lecture 5 compares DSPy vs. LangGraph agent framework approaches — stanfordnlp · 2026-11-19
- Mistral Large 4 coding test: 2 hours of agentic work burned just €0.82 of quota — qtnx_ · 2026-11-19
- Railway breach investigator: LLM attacker's infrastructure still targeting other AI companies — cyb3rops · 2026-11-19
- Gradio and HF launch hackathon: reproduce NeurIPS 2026 papers with AI agents — Gradio · 2026-11-19
- NVIDIA open-sources SkillEvaluator to vet AI agent skills before deployment — anthara_ai · 2026-11-19