HarnessEval: Evolving Evaluation from Metrics to Executable Systems
机器之心 · wechat · 2026-08-18
Systemic Upgrade in Evaluation Paradigm
MirroS, in collaboration with Tsinghua, Peking University, Berkeley, MIT, and others, released HarnessEval, proposing an upgrade from static metrics to an executable Agent evaluation system (Evaluation Harness).
Core Changes:
- From Rules to Workflows: Instead of running pre-set scripts, an Evaluation Agent understands the case, dynamically plans the path, and selects/combines skills.
- Four Stages:
- Plan: Understand case context and intent.
- Route: Select applicable skills from a library.
- Decompose: Break down abstract questions into verifiable sub-problems.
- Verify: Audit evidence and output scores.
Application: First launched for interactive World Model evaluation (HarnessEval-W). For complex issues like physical consistency and causal order in video generation, the system generates a hierarchical Evidence Tree to pinpoint error sources rather than just providing a scalar score.
Significance: Supports RSI (Recursive Self-Improvement). Evaluation becomes a key feedback component in the system evolution loop, not just an external ruler.
Related event: HarnessEval Turns Static Benchmarks into Agentic Evaluation Workflows(4 posts)→
More from Research
- UCSD Researchers Propose Dataset Watermarking Detectable via Black-Box API Access — yuxiangw_cs · 2026-10-16
- EmbodiedEvo tops RoboDojo as agents inherit reusable skills and lessons — Aiden_Tech_Ai · 2026-10-16
- CHI reviewer laments: papers' Discussion sections now cite nothing — IanArawjo · 2026-10-16
- Casper: Understanding 90% of neurons ≠ understanding the network — StephenLCasper · 2026-10-16
- NVIDIA Launches cuPotential in CUDA-X to Bring AI Agents to Semiconductor Materials Simulation — tomaszbednarz · 2026-10-16
- Agentic Search Systems Are Rewarded Only for Final Answers, Raising Faithfulness Concerns — burkov · 2026-10-16