ForgeSystem: An Architecture for Demand-Specific Artefact Production with AI Agents
Qingfeng He, Zhui Zhu, Yaojian Chen, Shangzhan Li, Leshan Li, Tingjian Zhang, Qian Xie, Yuxuan Li, Xu Han, Zhiyuan Liu
cs.SE
2026-11-09
ModelBest and Tsinghua's ForgeSystem has AI agents build demand-specific software: training engines reach 1.33x reference MFU, megakernels decode 1.21x faster than SGLang.
General-purpose software pays for its breadth with per-demand performance. Megatron-LM covers many models and hardware setups through one framework, so every specific configuration leaves performance on the table. DeepSeek's published numbers quantify the headroom: on a ten-thousand-GPU PCIe A100 cluster, the in-house HFReduce library sustained 6.3–8.1 GB/s inter-node allreduce bandwidth where NCCL delivered 1.6–4.8 GB/s, and the in-house framework roughly halved data-parallel and FSDP training time versus PyTorch. Specialisation pays, but human experts doing it per demand is too expensive to cover the long tail.
Two trends change the economics. Vendor-reported SWE-bench Verified resolution keeps climbing, and at fixed performance thresholds the per-question cost on GPQA Diamond has fallen from about $0.30 (o3, high) to the $0.0004 range for recent models. ModelBest and Tsinghua call the resulting paradigm Forge: agents construct a bespoke artefact for an explicit demand, and a forge launches only when expected specialisation benefit exceeds expected construction, verification and maintenance cost.
ForgeSystem has three layers.
The load-bearing choice is keeping verification out of production's reach. An agent optimising alone drifts toward test-gaming; here every performance number was earned under rules it cannot relax.
Three instantiations at different granularities.
| System | Baseline | Result |
| ForgeTrain | Megatron-LM / MindSpeed / PyTorch, 7 settings | MFU at 1.047–1.332× of reference; dense H100 models 1.05–1.09×, MiniCPM5 16B-A3B 1.177×, Ascend 910 settings 1.155× and 1.332× |
| ForgeMegakernel | SGLang 0.5.18 and others, 13 configs | MBU 50.5–85.9%, geometric-mean speedup 1.21× over SGLang, 1.54× over MPK; a 14th config reaches 74.0% MBU against a hand-written megakernel's 74.6% |
| ForgeKernel | FlashAttention-4, DeepGEMM, cuBLAS etc., 22 configs | 18 at ≥1.0×, 3 within 1%, 1 (sparse MLA prefill on B200) behind by 1.3%; geometric mean 1.016× |
| ForgeKernel | K-Search, CAKE and other generation systems, 7 configs | all ≥1.0×, geometric mean 1.394×, max 2.706× |
Quality parity is the more telling half. MiniCPM5 1B on Ascend finished the same 48,000-step decay schedule with final losses of 1.3227 (forged) versus 1.3226 (MindSpeed); average downstream score gaps across three comparisons were +0.68, −0.68 and +0.93 points. Decoding was checked against a float64 forward pass: on 200 GSM8K questions every configuration stayed within 1.5 percentage points of baseline accuracy, and serving decode latency came in below baseline everywhere, geometric mean 1.11×. The paper flags that 1.21× measures the isolated decode step while 1.11× reflects the benefit after service integration.
Effort and reuse: ForgeTrain needed about 12–20 rounds on H100 and 3–5 on Ascend to first beat its reference. ForgeMegakernel's median cost per kernel is 47 rounds, 14.7 agent-hours and US$177 (range $77–414). Re-forging still pays: GQA attention forward gained another 9.9% and then 4.2% over its own accepted output, about 15% above the expert library; DeepSeek sparse-attention forward gained 7.2%.
The contribution is less any single kernel than the packaging. VibeServe and Bespoke OLAP already generated one-off systems; ForgeSystem adds an explicit economic trigger for what to build, a verification wall between production and judgement, and an archive that compounds across forges. At a median of a few hundred dollars per kernel, amortisation works for long-running services where compute savings accumulate with traffic.
For ML infrastructure teams the quality numbers are the stronger signal: an agent-written training engine matched MindSpeed's final loss to within 0.0001 after a full schedule, and controlled comparisons show that explicit quality requirements steer agents toward optimisations that preserve training quality. The custom headroom per model-hardware configuration is real, and agents are a viable tool for mining it. The long-term argument deserves attention too: under Forge, the durable asset is the archive and production system behind the artefacts, not any single artefact, and whoever holds the accumulation controls production.
The authors concede the economic criterion fails when specialisation benefits little or verification and integration costs stay high (Appendix C). Beyond that: