LongWoF-Bench: Evaluating EvoMap Genes for Verifiable Long-Workflow Tasks
Xiao Zhang, Qumeng Sun, Jihao Li, Yiming Ren, Xiang Liu, Haoyang Zhang, Junjie Wang
cs.CL
2026-08-24
On 252 verifier-confirmed Opus trajectories in LongWoF-Bench, evolved Genes beat Skill by 8.7-15.5 points on seven models; reference-distilled Genes fall behind.
Models are now asked to finish whole workflows: interfaces, ordering, boundaries, consistency across files. Miss one constraint and the deliverable fails machine checks. A successful run is usually thrown away, so the next model rediscovers the same failure modes. A Skill can tell a model how the task is supposed to be done. EvoMap asks something else: once a verifier has accepted a trajectory, can that experience be stored as a structured Gene and reused.
Few benchmarks hold the task, runtime, and verifier fixed while comparing no extra context, Skill, and Gene side by side. LongWoF-Bench exists for that comparison.
A task is a public spec, an environment, an allowed deliverable space, and a verifier. Success depends only on whether the final artifact passes all mandatory checks. Verifiers are executable tests, hidden checks, or normalized exact match. Gold outputs, reference solutions, and verifier logic stay hidden; everything needed to succeed is still in the public spec.
The release has 778 machine-verifiable tasks: 341 code generation, 127 agent-environment synthesis, 151 math, 159 rule following. A typical agent-environment item is adaptive cruise control: public vehicle parameters, 150-second sensor traces, and performance limits; the solver must ship controller, simulator, and report; a private evaluator reruns the submission and checks interfaces and behavior.
Genes follow Evolver's execute-verify-revise loop. The producer tries blind first. On failure, the next rollout sees the previous solution plus sanitized verifier feedback, within a fixed budget. What gets revised is the solution, not the Gene. A passing trajectory is then distilled into strategies, prerequisite checks, boundaries, and failure guards. Claude Opus 4.8 produced 252 verifier-confirmed Genes. Evaluation uses three conditions (no context, Skill, EvoMap Gene) and seven consumer models from Anthropic, Gemini, MiniMax, and Qwen. One recorded trial per task-condition, with no verifier feedback at inference.
The main comparison is those 252 tasks. Mean strict pass rate across seven models: 41.0% with no context, 51.2% with Skill, 62.9% with Gene. Gene beats Skill on every model by 8.7 to 15.5 points, all significant under McNemar tests.
| Consumer | Skill | Gene | Delta |
| Claude Opus 4.8 | 63.9% | 79.4% | +15.5 |
| Claude Sonnet 4.6 | 55.6% | 66.3% | +10.7 |
| Gemini 3.1 Pro | 63.1% | 72.2% | +9.1 |
| MiniMax M3 | 50.4% | 63.5% | +13.1 |
| Qwen3.5-397B | 48.4% | 61.9% | +13.5 |
| Qwen3-Coder-30B | 31.8% | 43.2% | +11.5 |
| Gemini 3.1 Flash-Lite | 45.2% | 54.0% | +8.7 |
Opus itself passes 200 tasks with Gene versus 161 with Skill, 39 more, while solve-time tokens fall from 803,099 to 723,480, a 9.9% cut. Multi-round discovery of the 252 trajectories used 404 calls and 1,333,968 tokens; one-shot reuse uses 252 calls and 723,480 tokens, 45.8% fewer tokens. This is not an equal-success swap: discovery solves all 252, reuse solves 200.
Where the Gene comes from matters more than the Gene format. On the 526 tasks where Opus never got a passing trajectory in budget and a fallback Gene was distilled from reference-side teacher signals, Gene trails Skill on every model by 3.3 to 11.3 points. The two groups are different tasks, so this is provenance-associated evidence, not a same-task ablation. On the 180 tasks both Opus and Gemini solved, Opus-authored Genes beat Gemini-authored Genes for every consumer by 4.4 to 11.7 points.
Gains are uneven by family. Agent-environment and rule following are the most consistent, at +15.6 to +29.7 and +2.1 to +22.9 points. Code generation has three small regressions, 1.7 to 4.9 points. Math jumps for Opus from 43.3% to 83.3%, and is flat or worse for several others; Gemini Flash-Lite and Qwen3-Coder stay at 0 under all three conditions. A Gene can move formulas and boundary conventions. It cannot do the arithmetic for a model that cannot compute.
A Skill teaches procedure. A Gene teaches the exact semantics the verifier actually enforces. In the drill-hole assay task, a plausible "later interval wins" rule fails; the passing trajectory splits at every breakpoint, averages overlapping assays, then length-weights the bins. In emergency dispatch, a weapon on scene does not override the distance threshold, and equality does not trigger. Those are not generic workflow tips. They are the conventions that survive end-to-end checks.
For long-workflow agents, this is closer to externalized verified memory: pay the discovery cost once, amortize it across later models and runs. The trajectory has to have passed. Without a passing run, packing the same format into a Gene hurts.
The headline result lives on the 252 tasks Opus could already solve. On the 526 it could not, Gene is worse than Skill across the board. Each task-condition is a single trial, with no repeated-API variance. Discovery versus reuse is not an equal-success comparison, and the one-time cost of distillation and audit is omitted. Opus is the producer, so Genes may carry Opus-specific conventions. The benchmark and EvoMap come from the same lab; how carefully Skill was written is hard to audit from outside. Math shows the ceiling: transferred experience does not replace the consumer's ability to compute.