CyberFactory: Scaling Cyber Security Capabilities with Instances from the Wild
Jian Yang, Haau-Sing Li, Shawn Guo, Zixi Zhao, Yibo Tan, Jiajun Wu, Aishan Liu, Zhoujun Li, Xianglong Liu, Tianyu Zheng, Bryan Dai, Chengran Yang
cs.CR, cs.CL
2026-08-24
CyberFactory turns public CVEs into verifiable agent trajectories. OpenAegis hits 58.1% Pass@1 on CyberGym in one hour, 28.5 points above Qwen 3.5.
Closed-source models already show non-trivial skill at reproducing vulnerabilities in real codebases. Open-source work stalls in three places. Frontier open-weight models do not ship a reproducible training recipe. Public recipes mostly sit on isolated tasks such as CTF and do not scale agentic data. Bare rollouts also fail to pin down a long-horizon analysis procedure; that part needs a domain prior.
CyberFactory, from Beihang University, IQuest Research, and Singapore Management University, turns public vulnerability artifacts into executable, checkable supervision and trains OpenAegis on it.
The pipeline joins data construction, trajectory synthesis, and training across three tasks: PoC generation, patch generation, and cybersecurity question answering. A PoC here is a concrete input that triggers the target bug. Validation is differential: the input must fire a sanitizer failure on the pre-patch build and must not fire it on the post-patch build.
PoC instances come in three difficulty tiers. ARVO ships pre/post Docker images and a ground-truth PoC. OSS-Fuzz only names the introducing commit, so the pipeline binary-searches for the fix the same way ARVO does. Wild CVEs are the hard case: keep records with a CWE type, locate the fix commit from the affected version range, build both sides, then drop instances a model can already solve by emitting a PoC directly. Extra signals such as bug type or crash traces are used only during that filter and are discarded for synthesis and training.
Task text prefers a high-quality fix commit message; weak messages get a description built from the evidence and the fix. Patch instances follow the CVE-Factory construction from CVE records. QA is answer-first. The answer must come from a trusted source: execution facts, structural facts such as changed functions, or wording taken from an authoritative report. The LLM only rewrites the rest into a question, and a judge checks source traceability, no leakage into the prompt, and uniqueness.
During synthesis the teacher receives a reusable vulnerability-analysis skill: inspect the target and build constraints, explore candidate inputs with analysis and tests, validate evidence, and change course on failure. The skill is for data construction only. OpenAegis does not see it at inference. Trajectories that fail the task verifier are dropped. CyberGym-style rollouts can hit a 256K context wall. At 90% usage the trace is compacted into a continuation state that keeps verified evidence, failed attempts, open hypotheses, artifacts, build status, and pending actions, then resumes.
OpenAegis is full-parameter SFT from Qwen 3.5-397B-A17B for three epochs, packed to 131,072 tokens, with overlong examples discarded. Training uses BF16 on 256 GPUs with a peak learning rate of 2×10⁻⁵.
CyberGym gives only a natural-language description and a codebase. The agent must submit an input that reproduces the target bug. Every model gets one hour, the same scaffold, tools, and submission logic, and no skill at eval time.
| Model | Scale | Pass@1 |
| Qwen 3.5 | 397B-A17B | 29.6% |
| GLM 5.2 | 744B-A40B | 43.3% |
| Kimi K2.7 | 1T-A32B | 51.7% |
| OpenAegis | 397B-A17B | 58.1% |
That is +28.5 points on the same backbone, +14.8 over the larger GLM 5.2, and +6.4 over Kimi K2.7.
Compacting at 90% reaches 58.1% overall and 48.7% on long-horizon tasks (more than 40 tool calls), with a 7.0% context-exhaustion rate. Full history scores 52.1%/40.2%/18.7%. Naive truncation is worse. On long-horizon tasks, compaction beats full history by 8.5 points and cuts exhaustion by 11.7 points.
Giving GLM 5.2 the skill raises Pass@1 from 43.3% (one 60-minute try) to 46.5% (five 15-minute tries). The two runs are not compute-matched; the paper frames this as higher synthesis throughput. Exploration coverage jumps from 3.78% to 99.85%, validation coverage from 0.13% to 98.41%.
After SFT, OpenAegis still does not receive the skill, yet exploration calls per trajectory rise from 0.01 to 1.32 and validation from 0.00 to 1.05. Read calls fall from 28.4% to 7.3%, shell calls rise from 70.1% to 89.9%. Single-operation calls drop from 31.4% to 13.5%; calls with 6–10 operations rise from 4.3% to 30.8%. AddressSanitizer compile events go from 155 to 1,795. Trajectories that submit exactly once rise from 37.9% to 48.2%; those that submit five or more times fall from 10.4% to 2.0%.
The scarce piece for open security agents is often a verifiable data factory, not a bigger backbone. Wild CVEs become differential tasks. A skill pulls the teacher from ad hoc input guessing into prior-guided exploration plus evidence checks. SFT then writes that procedure into the weights, so inference does not need the skill.
There are two ways to use this. One is to run OpenAegis as a CyberGym-style reproduction agent. The more reusable piece is the recipe from public artifacts to executable instances to trajectories. The name Aegis is the shield of Zeus and Athena; the stated intent is defensive. The paper also flags that the weights carry risk and that intended use should be checked.
The headline result is a sizable specialization under a matched scaffold, not a new attack method.
The hard numbers are almost all on CyberGym PoC reproduction. Patch generation and CyberQA enter the training pipeline, but the paper says they still lack equally rigorous evaluation.
The skill comparison is not compute-matched: 60 minutes once versus 15 minutes five times. The teacher can also overfit the supplied prior. Some targets still respond better to hand-built inputs than to a fuzz-first strategy. Evaluation is bounded by available CVE artifacts, project coverage, and the one-hour budget. Training drops sequences longer than 131k tokens, which may cut the longest, hardest traces. Dual-use risk is noted in a footnote: a public, verifiable reproduction capability can be misused.