GDPevo: Evaluating Agent Self-Evolution on Real Business Tasks
Leijun Zhou, Zhihao Liu, Xiang Qu, Chenxu Liu, Yifei Liu, Yanke Yu, Jingzhe Xu, Xuejun Wu, Buyue Qian, Xi Chen, Yaowei Zheng, Junhao Hu
cs.AI
2026-08-04
GDPevo benchmarks agent self-evolution on enterprise workflows; rule hybridization makes gains attributable; the best agent rises 16.44 pp yet stays 24 pts below the 91.6% ceiling.
Agent self-evolution means an agent fixes what it learned from past tasks into some persistent state (a memory, a skill, a prompt) and reuses it on related tasks later. The hard part is not doing it; it is measuring it. How do you prove a gain on the test set came from the training experience, rather than the model's world knowledge or a task that was simply easy?
Two existing evaluation styles fall short. Evolution-native benchmarks mine train-test relationships from existing suites after the fact, and they tend to cover easy tasks with few such relationships. Evolution-adaptive benchmarks just slice a static task suite in half; because the split was never designed around transferable ability, any gain cannot be attributed to training. Both publish static public task sets, so data contamination can rot the scores at any time. GDPevo targets all three gaps at once: GDP-related enterprise tasks, attribution built into the design, and a fully automated pipeline to fight contamination.
The core mechanism is rule hybridization, and it exists to answer one question: was the gain actually learned? It works in three steps. First, decompose an enterprise scenario (a service ticket flow: verify account, diagnose fault, dispatch) into a set of atomic rules, each a minimal, independently checkable internal convention, such as one company's sponsor-status priority, another's blacklist exclusion, or a company-specific invoice expiration date. These rules are absent from the model's world knowledge, so the agent can only infer them from training. Second, scatter the rules so each of the five training tasks exposes only a subset. Third, recombine them across the five test tasks: one test task may invoke the priority rule and the blacklist rule together, even though no training task ever contained that combination. Only an agent that learned the rules and can compose them solves the test, which makes the gain attributable.
The evaluation itself is built to make the numbers trustworthy. Every task is scored by a deterministic rule-based grader: an LLM turns each rubric point into a code-based test case rather than acting as a judge, so runs are reproducible and each failure traces to a specific violated rule. Cost (tokens, turns, money) is reported alongside accuracy. Candidate task groups pass a calibration gate: solving the test tasks with no training should land in the 40 to 60 percent band (too easy or too hard is rejected), fewshot after evolution should add another 0.1 to 0.3, and the final score must stay below 0.8 to leave headroom. Six independent reviewer agents vote, and a group is accepted only if at least five agree.
An agent is defined as Harness + Model. The study runs two harnesses (Codex, Claude Code) with four models (GPT-5.5, Opus-4.8, GLM-5.2, DeepSeek-V4-Pro-Preview). There are four supervision types: base (no training), fewshot (questions plus gold answers, analogous to supervised fine-tuning), reflect (questions plus the score of the agent's own attempt, run for three rounds, analogous to reinforcement learning), and self (questions only, no answers, analogous to unsupervised learning). The evolution method is uniformly skill-based: experience is distilled into a SKILL.md artifact carried into testing. The benchmark has 240 tasks in 24 groups across six domains (CRM, ERP, finance, healthcare, legal, data-centric), with five training and five test tasks per group.
| Agent (harness / model) | base acc | fewshot acc | gain |
| GPT-5.5 / Codex | 49.37% | 64.51% | +15.14 pp |
| Opus-4.8 / Claude Code | 50.63% | 67.07% | +16.44 pp |
| GLM-5.2 / Claude Code | 46.12% | 60.09% | +13.97 pp |
| DeepSeek-V4-Pro-Preview / Claude Code | 43.58% | 48.79% | +5.21 pp |
Fewshot is the most reliable supervision type, and every evolved configuration beats its base, by up to 16.44 pp. The key contrast is the oracle ceiling: feed the model all hidden rules plus the gold answers so it applies rather than learns, and it reaches 91.6%. The best evolved configuration (Opus fewshot at 67.07%) sits about 24 points below that. The self-evolution ability of current agents is far from realized.
The cost numbers are the surprise. GPT-5.5 fewshot raises accuracy by 15.14 pp while cutting test-time cost by 20.88%. A cheaper model plus evolution can approach or beat a bigger model running bare: DeepSeek-V4-Pro-Preview fewshot reaches 48.79% at about 1/28 the cost of GPT-5.5 base (49.37%), and GLM-5.2 fewshot actually surpasses GPT-5.5 and Opus base by 10.72 and 9.46 pp at roughly half the cost. A counterintuitive result: a weaker starting point does not mean more evolution headroom. DeepSeek has the lowest base (43.58%) yet the smallest gain (+5.21); Opus has the highest base (50.63%) and the largest gain (+16.44).
The cross-domain transfer experiment covers only CRM, ERP, and finance (to keep it tractable). The diagonal (train and test in the same group) is always positive. Fewshot overfits the source domain like supervised fine-tuning: five of six off-diagonal cells are negative, worst case -5.0 pp (train on finance, test on ERP). Reflect behaves more like reinforcement learning, with milder overfitting: half the off-diagonal cells are positive, best case +6.5 pp (ERP to finance), and the worst case loses only 1.0 pp.
RQ3 asks which matters more, the evolution method (the skill creator) or the model. Holding the agent fixed and varying only the creator across five options (a one-sentence Naive prompt the authors wrote, plus the built-in creators of Claude Code, Codex, OpenCode, and deepagents), Naive performs on par with or better than the elaborate off-the-shelf creators. The magnitude of evolution comes mainly from the model's own intelligence, not from the guidance encoded in the method, and over-engineering the method can even hurt.
For anyone building agents, this benchmark hands over hard data on whether evolution is worth doing. The answer is yes, and it is cost-effective: a cheap model with fewshot skill evolution can approach or beat a much larger model running bare, at far lower cost. It also offers a reusable recipe against contamination, since the fully automated pipeline expands the suite from V1's 120 tasks to V2's 240 in two days, so a leaked public set can simply be regenerated. Deterministic grading without an LLM judge, plus attribution built into the train-test split, turns the score into a measurement instrument rather than a number of unclear origin.
Its position is also clear: this is a methodology and benchmark paper with no new model, and its heaviest conclusion is that current agents still self-evolve poorly. For anyone choosing an agent stack, building an evaluation, or estimating the agent capability ceiling, it is one of the few works that ties real enterprise tasks to attributable measurement.
The authors concede several. First, the 91.6% oracle ceiling is approximate: they lack human domain experts across legal, medical, and financial workflows, so they cannot measure "how well experts do when handed all the rules," and approximate the ceiling with "feed the model all the rules" instead. The number reads more as the upper bound when rules are fully transparent, not as a human-expert ceiling or necessarily a target agents can approach. Second, every experiment uses a single evolution method (skill-based). The authors stress the benchmark is agnostic to the form of persistent state, but the RQ3 claim that "the model matters more than the method" holds only under this one method and should not be over-generalized. Third, the cross-domain transfer experiment takes only three domains with one group each, so the sample is small and the paper does not report statistics on values like -5.0 and +6.5.
Two further concerns stand out. Reflect transfers more robustly across domains than fewshot, yet within a single group fewshot usually scores higher (the paper calls fewshot the most reliable, and on Opus fewshot reaches 67.07 versus reflect-3 at 59.27); the tension between these two findings is never reconciled. The calibration gate (base must land in 40 to 60 percent) screens tasks using one agent, and the paper does not discuss whether the rejected tasks behave consistently across models, which could introduce a latent selection bias.